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(54) KEY SIGNAL GENERATING DEVICE, PICTURE PRODUCINQ DEVICE, KEY SIGNAL 
GENERATING METHOD, AND PICTURE PRODUCINQ METHOD 

(57) In this invention, when the portion from which a 
key signal a is to be generated of picture is caused to be 
foregrourKi F. the portion except for the above is caused i 
to be background B. and pixel value of pixels constitut- 
ing that picture is designated at C. the key signal a is 
generated so as to satisfy the relational expression 
C 3 (F-B)a' f (prime) represents differentiation). In 
more practical sense, a change detecting section 10 
cakxilates change C of pixel value between pixels con- 
stituting picture, and an F-B detecting section 30 calcu- 
lates difference F-B between the foreground and the 
backgrowTd. In addition, an integral value calculating 
section 40 divides the change C by the difference F-B 
to integrate the result obtained by divisional operation to 
thereby determine key signal a. As a result precise key 
signal can be obtained in this inventkxi. 
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Description 
Technical Field 



This invention relates to a key signal generating apparatus and a picture synthesis apparatus, and a key signal gen- 
erating method and a picture synthesis method. More particulariy, this invention relates to a key signal generating appa- 
ratus and a picture synthesis apparatus, and a key signal generating method and a picture synthesis method which are 
adapted for detecting change of pixel value between pixels constituting pictures to generate a key signal on the basis 
of the change of the pixel value, thereby making it possible to obtain an accurate key signal. 



Background Art 



For example, in movies and other image productions, in the case where various simulations of pictures (images) 
are earned out or any special effect is given to pictures, a remarked portion (object) is extracted from picture, or the 
extracted portion is synthesized with respect other pictures. Such a partial extraction from picture and/or synthesis (of 
the extracted portton) is can'ied out by using key signal. 

As a technology relating to such a key signal, an exanple thereof is proposed in the specification and the drawings 
of the U.S. Patent Na 4.488,169 (Registered Date: December 1 1 . 1984) by the applicant of this invention. 

As the key signal, there are a key signal called hard key and a key signal called soft key Now. in the case where 
e.g.. the remarked portion within picture used tor generating the key signal is caused to be the foreground (foreground 
picture), and the portion except for the above is caused to be the background (background picture), a binary key signal 
(mask picture) in whfch the range of the foreground in the picture is caused (assumed) to be 1 and the range of the 
background is caused (assumed) to be 0 as shown in FIG. 1A is called hard key. On the contrary, a key signal which 
can take not only two values of 0 and 1 , but also continuous real nunt)ers within the range from 0 to 1 as shown in FIG. 
25 1 B in consideration of alias and/or motran blur which will be described later taking place in picture is called soft key. 
Here, the area where the key signal takes values within the range from 0 to 1 is called gray area as the occasion may 
demand. 

Namely, it can be said that the hard key is a signal having a sharp boundary (border) line (signal in which inclination 
(gradient) of the boundary line where the key signal changes from 0 (or 1) to 1 (or 0) is sharp), and the soft key is a 
30 signal having a smooth (gentle) boundary (bonder) line (signal in which inclination (gratfent) of the boundary line where 
the key signal changes from 0 (or1) to 1 (or 0) is smooth (gentie)). 

It is now assumed that a (a value) shown in FIGS. 1 A and 1 B indicates continuous key signal values and also indi- 
cates, in this case, values of the key signal of pixel units at the same time. In practice. e.g.. continuous key signals are 
caused to undergo filtering by the range of pixel, etc. to use. as the key signal of the pixel unit, the result by that filtering 
at the center of corresponcfing pixel. In this case, in the picture, pixel value of the pixels at the boundary portion between 
the foreground and the back^ound is a value in whfeh respective signal components of the foreground and the back- 
ground are interposed. Rom fads as above, it can be said that a indicates contribution ratio (factor) indicating to what 
degree the foreground contributes with respect to respective pixels. 

Meanwhile, it is rare that in the case where both the foreground and the background are stationary, the boundary 
between the foreground and the background is positioned at the boundary line between pixels constituting a con-e- 
sponding picture. Ordinarily, as shown in FIG. 2A. the boundary between the foreground and the background exists 
within the pixel having finite dimensions. However, since the pixel is the minimum unit constituting the picture, even if 
the boundary between the foreground and the background as described above exists within the pixel, that pixel is 
caused to have color or brightness (pixel value) of any one of the foreground and the background as shown in FIG. 2B. 
45 For this reason, the boundary line between the foreground and the background is drawn along the boundary line 
between pixels as shown in FIG. 2C. When the entirety of such a picture is viewed, unnatural flicker, i.e., alias takes 
place at the portfon of the boundary between the foreground and me background (boundary line between pixels). 

Accordingly, even if a key signal smoothly indicating the boundary line between the foreground and the background 
can be obtained, alias would take place in the synthetk: picture obtained as the result of the fact that the key signal men- 
50 tioned above is used to carry out picture synthesis. 

In view of the above, as a method of preventing alias taking place in the synthetic picture, there are, e.g.. methods 
disctosed in the Japanese Patent Application Laid Open No. 232787/1985 and the Japanese Patent Application Laid 
Open No. 340671/1992, In the case of the method disclosed in the Japanese Patent /^plication Laid Open No. 
232787/1 985, the hard key is used to carry out picture synthesis, and to apply low-pass filter (LPF) only to the boundary 
55 portion between the foreground and the background to allow the boundary portion thereof to be blun-ed to allow it to be 
smooth to look at. In this method, since the hard key is used, but LPF is applied to the boundary portion between the 
foreground and the background, it can be said that picture synthesis is carried out by using key in which the key signal 
is caused to have gradient Ondination). i.e.. substantially soft key. 

Moreover, in the method disclosed in the Japanese Patent AppUcation Laid Open No. 340671/1992, a block in 
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which a remarked pixel on the boundary between the foreground and the background is caused to be the center is set 
to calculate a key signal (soft key) from the pixel pattern constituting the foreground and the background within that 
blo^ to carry out. by using such a key signal, extraction of the foreground, and synthesis of the foreground and the 
background to thereby reduce the alias. 

However, since the soft key used in the above-mentioned method is such that value of a at the inclined (gradient) 
portion (the porton of 0 < a < lln FIG. (B)). i.e.. in the gray area is unifonn. there was the problem that it is difficult to 
cope with motion blur taking place, e.g., in the case where the foreground is moving. 

TTie motion blur will now be briefly described. The moving picture is realized as the result of the fact that picture 
(stll picture) IS continuously displayed, e.g.. in frame units, etc. For example, assuming now that a circular object mov 
«r^ from the left direction to the right cfirection within picture exists, in the case where the frame period is assumed to 
be a short time such that it can be considered to be infinitesimal (infinitely small) with respect to the movement velocity 
Of that object, that object is displayed as a circular object in respective frames as shown in FIG. 3A. However, in the case 
where the frame period is not a short time such that it can be cwisidered to be infinitesimal with respect to the move- 
ment veloaty of the circular object, picture of frame at a certain time tg becomes a picture in whfch movement of the 
Object from time t^ of frame preceding by one to the time tg is reflected as shown in FIG. 3B. Namely, the object within 
the picture of the frame at the time tg is not circular picture image, but picture image extended (elongated) in a thin man- 
ner in the direction of that motion such that the contour is Wur. Such a phenomenon is called motion blur. 

It Is to be noted that while motion blur takes place not only in the case where any one of tine foreground and the 
background is moving, but also in the case where botii are Independently moving, in the case where only the back- 
ground ts moving and in ttie case where botii are independentiy moving, it is possible to consider ttiose cases similarly 
to ttie case where only the foreground is moving if the background is taken as reference. In addition, in tfie case where 
motion Wur takes place, dear boundary between ttie foreground and ttie background does not exist. 

As a mettiod of generating a key signal in whfch motion blur Is taken into conskieration as desaibBd above tfiere 
are metfiods, e.g.. ttie Japanese Patent Application LakJ Open rkx 153493/1993 and ttie Japanese Patent Application 
Lakl Open Na 236347/1993. In the mettiod disclosed in ttie Japanese Patent Application Laid Open No. 153493/1993, 
current frame and frame preceding (earlier) by one are compared witti each ottier to ttiereby determine sum total witti 
respect to ttie entirety of frame of change quantities (magnitudes) of pixel value of pixels constituting ttie foreground of 
the cun-ent frame (a ttie frame preceding by one) and pixel value con-esponding to pixels of ttie frame preceding by one 
(or the current frame) to determine ttie gray area of ttie soft key from ttie sum total valua However, in ttiis mettiod, since 
ttie soft key is determined in dependency upon ttie above^iesaibed change quantity of pixel value (difference between 
pixel values of the same pixel), ttie direction of movement of ttie foreground is not taken into conskleration. so whettier 
or not con-ect soft key corresponding to ttiat movement can be obtained was not clear. 

On ttie ottier hand, in the mettiod disclosed in ttie Japanese Patent Application Laid Open No. 236347/1993. an 
approach is employed to first generate hard key to shave, at respective points on tiie boundary between ttie foreground 
35 and ttie background, the side surface of ttiat hand key (side surfaces of column in which ttie portion of a « I is caused 
to be ttie bottom surface shown in FIG. 1 A) by ttie elliptical cone to ttiereby generate soft key. In ttiis case, the tong axis 
and ttie short axis of ttie ellipse constituting ttie bottom surtee of ttie elliptical cone is determined by motfon vecta of 
the remarked portion and the magnitude of ttie edge intensity. However, in ttiis method, ttie direction of ttie edge inten- 
sity is not correctiy taten into oonskleration. For ttiis reason, whettier or not precise or con-ect soft key corresponding 
40 to movement of ttie foreground can be obtained was not clear similarly to ttie above. 

This invention has been made in view of such circumstances and can contenplate generating a key signal which 
is precisely coping witti motion blur, and pennits ttie influence of alias to be reduced. 

Disclosure of ttie Invention 

45 

A key signal generating apparatus according to tiiis invention comprises change detecting means for detecting 
change of pixel value between pixels constituting pwture. and key signal generating means for generating a key signal 
on ttie basis of ttie change of ttie pixel value oulputted from ttie change detecting means. 

A pkAjre synttiesis apparatus according to ttiis invention comprtees change detecting means for detecting change 
50 of pixel value between pixels constituting a first picture, key signal generating means for generating a key signal on ttie 
basis of the change of ttie pixel value outputted from ttie change detecting means, and syntiiesis means fbr synttiesiz- 
ing foreground picture of ttis first picture and background picture of a second picture by using ttie key signal generated 
by ttie key signal generating means. 

A key signaJ generating method according to ttiis invention comprises ttie steps of detecting change of pixel value 
55 between pixels constituting picture, and generating a key signal on ttie basis of ttie change of the pixel value. 

A picture synthesis mettiod according to ttiis invention comprises ttie steps of detecting change of pixel value 
between pixels constituting a first picture, generating a key signaJ on the basis of the change of ttie pixel value, and syn- 
ttiesizing foreground picture of ttie first pkrture and background picture of a second picture by using ttiat key signal. 

In ttie key signal generating apparatus according to ttiis invention, ttie change detecting means detects change of 
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pixel value between pixels constituting picture, and the key signal generating means generates a key signal on the basis 
of the change of the pixel value outputted from the change detecting means. 

In the pknure synthesis apparatus according to this Invention, the change detecting means detects change of pixel 
value between pixels constituting a first picture, and the key signal generating means generates a key signal on the 
basrs of the change of the pixel value outputted from the change detecting means. In addition, the synthesis means syn- 
thesizes foreground picture of the first picture and background picture of a second picture by using the key signal gen- 
erated by the key signal generating means. 

In the key signal generating method according to this invention, change of the pixel value betweai pixels constitut- 
ing picture IS detected to generate a key signal on the basis of the change of the pixel value. 

In the picture synthesis method according to this invention, change of pixel value between pixels constituting a first 
picture IS detected to generate a key signal on the basis of the change of the pixel value to synthesize foreground pic- 
ture of the first picture and the background picture of a second picture by using the key signal thus generated. 

Brief Description of the Drawings 



FIG. 1 A is a view for explaining hard key. and FIG. 1 B is a view for explaining soft key. 
FIGS. 2A. 2B and 2C are views for explaining alias. 
FIGS. 3A and 3B are views for explaining motion blur. 

FIG. 4 Is a view showing the state where local texture of picture is considered to be uniform. 

FIG. 5 is a view for explaining a method of calculating picture in which motion blur takes place. 

FIG. 6 is a view showing ffltering (filter processing) for providing the effect of motion blur 

FIG. 7 is a view showing foreground F and key signal a in which the effect of motion blur is taken into consideration 

FIG. 8 is a view showing that local texture of the foreground F to which the effect of motton blur has been applied 
is considered to be unifonm. 

FIG. 9 is a view showing that alias takes place in the case where pixel has finite size (dimensions). 

FIG. 1 0 is a block diagram showing the configuration of a first embodiment of a key signal generating apparatus to 
which this invention is applied. 

FIG. 11 is a flowchart for explaining ttie operation of the key signal generating apparatus of Fia 1 0. 

FIG. 12 is a block diagram showing an exanple of the configuration of change detecting unit 10 of FKa. 10. 

FIG. 13 is a ffowchart for explaining the operation of the change detecting section 10 of Fia 12. 

FIG. 1 4 is a view showing sobel operator. 

FIG. 1 5 is a view for explaining processing of step SI -2 of FIG. 13. 

FIG. 1 6 is a block diagram showing an example of the configuration of feature point detecting section 20 of FIG. 1 0. 

FIG. 1 7 is a view for explaining processing of ttie feature point detecting section 20 of Fia 16. 

FIG. 18 is a ftowchart for explaining the operation of the feature point detecting section of Fia ia 

FIG. 1 9 is a block diagram showing an example of F-B detecting section 30 of FIG. 10. 

FIG. 20 is a view for explainir^ processing of very small area detecting section 31 of FIG. 1 9. 

FIG. 21 is a view for explaining processing of difference calculating section 34 of FIG. 19. 

FIG. 22 is a flowchart for explaining the operation of the F-B detecting section 30 of FIG. 19. 

FIG. 23 is a view for explaining processing of step S5 of FIG. 1 1 . 

FIG. 24 is a block tfagram showing an example of the configuration of integral value calculating section 40 of FIG 

10. 

FIG. 25 is a view for explaining processing of integral value calculating section 40 of FIG. 24. 
FIG. 26 is a flowchart for explaining the operation of ttie integral value calculating section 40 of Fia 24. 
FIG. 27 is a block diagram showing an example of the configuration of key signal generating section 50 of FIG. 10. 
FIG. 28 is a view lor explaining processing of the key signal generating section 50 of FIG. 27. 
FIG. 29 is a ftowchart tor explaining the operation of ttie key signal generating section 50 of FIG. 27. 
FIG. 30 is a block diagram showing the configuration of a second embodiment of ttie key signal generating appa- 
ratus to which this invention is applied. 

FIG. 31 is a btock diagram showing an example of the configuration of feature point detecting section 60 of Fia 30. 
FIG. 32 is a view for explaining processing of line thinning section 61 of FIG. 31 . 

FIG. 33 is a btock diagram showing a third embodiment of the key signal generating apparatus to which this inven- 
tion is applied. 

FIG. 34 is a view for explaining processing of integral value calculating section 70 of FIG. 33. 
FIG. 35 is a flowchart for explaining ttie operation of ttie key signal generating apparatus of FIG. 33. 
FIG. 36 is a btock diagram showing an example of ttie configuration of the integral value calculating section 70 of 
FIG. 33. 

FIG. 37 is a flowchart for explaining ttie operation of integral value cateulating section 70 of FIG. 36. 

FIG. 38 is a btock diagram showing ttie configuration of a first embodiment of a ptoture synthesis apparatus to 



20 



EP0 771 107 A1 

which this invention is applied. 

FIG. 39 is a block diagram showing the configuration of a second embodiment of the picture synthesis apparatus 
to which this invention is applied. 

FIG. 40 is a block diagram showing an example of the configuration of information generating unit 90 of FIG. 39. 

Best Mode for Can-ying Out the Invention 

Prior to description of prefen-ed embodiments of this invention, in order to clarify the correspondence relationship 
between respective means of the invention desaibed in the claims and the en*odiments desaibed below, the features 
of this invention will be described in the following manner in the state where the description of corresponding embodi- 
ments (which should be considered to be as examples) are added within parentheses aftw respective means. 

Namely, a key signai generating apparatus according to this invention is directed to a key signal generating appa- 
ratus adapted for generating a key signal from picture, the apparatus comprising: change detecting means (eg 
change detecting section 10. shown in FIG. 10. 30 or 33. eta) for detecting change of pixel value between pixels con- 
stituting picture; and key signal generating means (e.g.. integral value calculating section 40 shown in FIG. 10 or 30. or 
integral value calculating section 70 and key signai generating section 50 shown in FIG. 33, etc.). 

The key signal generating apparatus according to this invention further comprises difference detecting means (e.g. . 
F-B detecting section 30 shown in FIG. 10 or 30. etc.) fbr detecting difference between foreground picture and back- 
ground picture constituting the picture to output difference information indicating the difference, wherein the change 
detecting means outputs change information indicating change of pixel value. arKi the key signal generating means 
generates the key signal on the basis of ti^e difference information and the change information. 

In the key signal generating apparatus accading to this invention, the key signal generating means comprises nor- 
malizing means (e.g., normalization executi'ng section 42 shown in FIG. 24. etc.) for normalizing the change infbrmation 
by the dfference Information to output normalized information obtained as ttie result of ttie normalization, and integra- 
25 tion executing means {e.g., integration executing section 43 shown in FIG. 24. etc.) for Integrating the normalized infer- 
mation afong a predetermined integral path). 

In the toy signal generating apparatus according to ttiis invention, the key signal generating means comprises 
range detecting means (e.g.. integnal path detennining section 71 shown in FIG. 36. etc.) for detecting integral range, 
definite integral information generating means (e.g., definite integral value calculating section 72 shown in FIG. 36. etc.) 
30 for integrating the change information along a predetermined integral path within the integral range to generate definite 
integral information of the change information within the integral range, normalizing means (e.g.. normalization execut- 
ing section 73 shown in Fia 36, etc.) fbr normalizing the change information by ttie definite integral infbrmation to out- 
put normalized infomiation obtained as the result of ttie normalization, and integration executing means (eg., 
integration executing section 74 shown in FIG. 36. etc.) for integrating tiie normalized infbrmation to ttiereby generate 
35 a key signal. 

A picture syrthesis apparatus accading to ttiis invention is directed to a picture synthesis apparatus adapted for 
receiving a first picture consisting of at least foreground picture and background pfoture. and a second picture consist- 
ing of at least background pkmjre to synthesize the foreground pfoture of ttie first picture and ttie background picture of 
ttie second picture, ttie apparatus comprising: change detecting means (e.g.. change detecting section 10 shown in 

40 FIG. 10. 30 or 33. etc.) for detecting change of pixel value between pixels constituting ttie first picture: key signal gen- 
erating means {e.g.. integral value cateulating section 40 shown in FIG. 10 or 30. or integral value cateulating section 
70 shown in FIG. 33. etc.) for generating a key signal on the basis of ttie change of ttie pixel value outputled from ttie 
change detecting means: and synttiesis means (ag., mixer 65 shown in FIG. 36 or mixer 96 shown in FIG. 39. eto.) for 
synttiesizing the foreground picture of ttie first ptoture and the background picture of ttie second picture by using the key 

45 signal generated by ttie key signai generating mean& 

"Rie picture synttiesis apparatus according to ttiis invention further conrprises difference detecting means (e.g.. F- 
B detecting section 30 shown in FK3. 10 or 30. eto.) for detecting difference between ttie foreground picture and ttie 
background pidure of ttie first pteture to output difference infbrmation indrcating ttie difference ttius detected, wherein 
the change detecting means outputs change infomnation indicating ttie change of ttie pixel value, and ttie key signal 

50 generating means generates ttie key signai on ttie basis of ttie difference informatkjn and ttie change infbrmation. 

It shouU be noted ttiat it is a matter of course ttiat ttie foregoing desaiption does not mean tiiat ttie respective 
means are limited to the above-mentioned conponents. 

The ttieoretical ground (principle) of ttiis invention will now be described. It is to be noted ttiat picture is assumed 
to be one-dimensional picture here for ttie brevity of explanation: 

55 It is now assumed ttiat pixel value of pixel at position (coordinate) x constituting one-dimenstonal picture C. More- 
over, ttie picture C is assumed to consist of foreground (foreground picture) F and background (background picture) B. 
and pixel values of pixel at ttie position x c<»istituting ttie foreground F and ttie backyound B are respectively desig- 
nated at F(x) and B(x). It is furttier assumed ttiat value at the position x of ttie key signal for extracting ttie foreground F 
from ttie picture C (contribution rati of ttie foreground F at the position x) is a(x). In ttiis case, it is assumed ttiat a(x) 
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takes a value within the range from 0 to 1 . and respectively takes 1 and 0 at positions of pixels completely constituting 
the foreground F and the background B. In addition, at the positions of pixels where the foreground F and the back- 
ground B are mixed. a(x) is assumed to take a value corresponding to the degree of contribution of pixel value of the 
foreground F. 

. In this case, the pixel value C{x) of pixels constituting the picture C is considered to be value in which the pixel value 
F(x) of pixels constituing the foreground F and the pixel value B(x) of pixels constituting the background B are synthe- 
sized at a ratio conresponding to the key signal a(x). Accordingly, the pixel value C(x) can be expressed by the fbllowina 
equation (1): 

C(x)«a(x)F(x) + (l-a(x))B(x) (1) 

Here, the previously desaibed alias takes place because pixel has definite size (dimensions), and motion blur 
takes place in the case where any one of the foreground F and the background B is moving, and in the case where both 
are independently moving. Explanation will now be given in a manner classified into four cases: 

tl] Case where the fbreground F and the background B are stationary, and pixels can be considered to be infinitely 
small. 

[2] Case where the foreground F or the background B is moving, and pixels can be considered to be infinitely small. 
[3] Case where the foreground F and the background B are stationary, and pixels have definite size (dimensions). 
14] Case where the foreground F or the background B is moving, and pixels have definite size (dimensions). 

[11 Case where the foreground F and the background B are stationary, and pixels are considered to be infinitely small. 

In this case, since the fbreground F and the background B are stationary, no motk)n blur takes place. Moreover, 
since pixels are considered to be infinitely small, no alias also takes plac& Accordingly, the key signal a(x) takes any 
one of value of 0 and 1 . and the position x where that value becomes equal to 1 is in correspondence with the existing 
range of the fbreground F. 

Here, when the equation (1) is differentiated by x. the equation (2) is obtained. It should be noted that description 
of argument (x) of the function is omitted as the occasion may demand. 

C' = (F.B)a' + (P-B')a + B* (2) 

In the above equation. ' (prime) represents differentiation. 

In the case where ordinary picture is assumed, texture (pixel value) of the foreground F and the background B is 
not uniform, but varies as shown in FIG. 4. However, in the case where the foreground F and the background B are 
locally viewed. e.g.. in pixel units, etc.. that texture can be considered to be uniform as shown in the FIG. 4 mentioned 
above. Accordngly, from a local point of view, tiie pixel value of tiie fbreground F and the pixel value of the background 
B can be considered to be constant As a result, the differential value F and the differential value B* can be both con- 
sidered to be zero (0). Rom this fact, the equation (2) can be rewritten into the ibilowing equation (3): 

C.{F-B)a' (3) 

Since the differential value C of the left side of the equation (3) is a value obtained by differentiating pixel value C(x) 
constituting the picture C by x. this differential value C represents change of pixel value between pixels constituting the 
picture C. It is seen from the equation (3) that change C'(x) of pixel value between pixels of the picture C and differential 
value a'(x) of the key signal have the proportional relationship in which difference F(x)-6(x) between pixel value F(x) of 
the foreground F and pixel value B(x) of the background B is caused to be propational constant 

[2] Case where ttie foreground F a the background B is moving and pixels can be considered to be infinitely small. 

In the case where attention is now drawn to picture of moving object, nrtotion blur takes place in the picture of the 
curent frame as has been explained with refererKe to FIG. 3. Namely, ttie picture of the current frame is the picture in 
which movement of ttie object from the frame prececfing (earlier) by one relative to ttie current frame up to the cun-ent 
frame is reflected. Accordingly, in the case where times of the current frame and the frame preceding by one are respec- 
tively designated at t2. ti . and pixel value at time t of pixel at a certain position x of the picture C is represented by Ct(x). 
pixel value Ct2(x) of pixel of the position x at the time t2 can be represented by time integration of picture Sx(t) which 
has been passed through the pixel of the positton x for a time period from the times t^ to tg. Namely, ttie pixel value 
Ct2(x) can be expressed by the following equation (4): 
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CX2{x) ^ jSx[t)dt 



However, there ts no way to find out (recognize) the picture Sx(t) which has been passed through the pixel of the 
positon X for the tme period from times t, to tg from the picture of the current frame, or the picture of the frame preced- 
mg by one. In view of the above, whai the picture located at the position x is represented by SUx) at the time to of the 
current frame, pictures existing within the range from the positions x, to are respectively represented by pictures 
St2(xi) to St2{x2). In this case, when it is assumed that the pictures StgCxi) to St2(X2) have been passed through the 
pixel located at the position x for the time period from the times ti to tg. whereby pixel value CtgCx) of picture of the posi- 
hon X at the time tg has exp^ienced the influence of the pctures St2(x,) to StzCxg). the pixel value Ct2(x) is equal to time 
integration of the picture Sx(t) which has been passed through the pixel of the position x for the time period from the 
times ti to t2 as shown in FIG. SB. and can be represented by the spatial integration of the pictures St2(x,) to Stofxo) 
existing within the range from the positions x^ to xg at the time tg. Namely, the pixel value Ciolx) can be expressed by 
the following equation (5): . ^ j 
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Ct2(x)» JSfgWdx (5) 

Meanwhile, when still picture (picture in which no motion blur takes place) at the time t2 is represented by S(x) and 
filter for giving the effect of the motion blur with respect to the stiU picture is represented by R(x). the right skie of the 
25 equation (5) Is equal to convolution integral of S()0 and R(x) as shown in FIG. 6. Accordingly, the equation (5) can be 
rewritten as indicated by the following equation (6). 

Ct2(x)oS(x)-R(x) (6) 

30 In tfie above equation. • indicates convolution integral. 

Now. filters which give the effect of the motion blur to the foreground F and the background B are assumed to be 
respectively represented by Rf(x) and Rb(x). In addition. In this embodiment since the key signal a(x) moves atong with 
the foreground F. in the case where the efftet of the motion blur is given by ttie filter Rf (x) with respect to the foreground 
F. rt is necessary to give the effect of the motion blur also to the key signal a(x) bltr by the same filter Rf(x) as shown in 
FIG. 7B. When the effect of the motton Wur is given to the foreground F. the background B and the key signal a in con- 
sideration of the above-mentioned facts to rewrite ttie equation (1). the following equation (7) is provided: 



35 



C « {a*Rf){F*Rf) + (1 .a*Rf){B*Rb) (7) 

40 Differentiation Of the above equation (7) gives the fdtowing equation (8): 

C* » (F*Rf . B*RbKa*Rf) + (F*Rf - B**Rb)(a-Rf) + B-'Rb (8) 

If the texture of tfie picture of the foreground F to whteh the effect of the motfon Wur has been given, i.e.. the texture 
45 Of the prcture obtained by filtering the foreground F by the f IHer Rf can be considered to be also uniform similarly to the 
case shown in FIG. 4 from a local point view. This similarly applies to the pixel value B of ttie background B. 

Accordingly, from a focal point of view, the pixel values F^Rf and B*Rb which have been caused to undergo filtering 
can be considered to be constant. As a result, these differential values (F*Rf)' and (B*Rb)* can be both considered to 
be zero (0). from this fact the above-mentioned equation (8) can be rewritten as indicated by the fWtowing equation (9): 



so 



C*«(F*Rf.B*«>)(a-Rf)' (9) 



It is seen from the equatfon (9) that change C of tiie pixel value between pixels of the picture C and differential 
value (a *Rf)' of the key signal in whidi the effect of the motion Uur is taken into oonskieration have the proporttonal 
55 relationship In which difference F'Rf-B'Rb between PRf of the foreground and the pixel value B*Rb of the background 
in whfoh the effect of the motion Wur is similarly taken into consideration is caused to be proportional constant. 
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[3] Case where the foreground F and the back ground B are stationary, and pixels have definite size 

In the case where the size of pixel is considered to be Infinitely small, the pixel value smoothly varies with respect 
to the position x as shown in FIG. 9A. (n contrast in the case where pixels have definite size, the pixel value varies in a 

5 step form with respect to the position x as shown in FIG. 9B. For this reason, in the case where the signal shown in FIG 
9A IS merely sampled, alias takes place at the boundary between the foreground F and the background B as previously 
desaibed. In order to reduce such alias, there is a method, as shown in FIG. 9A. in which, for the purpose of calculating 
the pixel value, the pixel value is caused to undergo filtering by the lowijass fitter G{x) to carry out sampling i e value 
obtained as the result of convolution integral of the pixel value and the low-pass filter G(x) at the pixel position is caused 

10 to be new pixel value. In view of the above, in the case where picture in which alias is reduced by the low-pass filter G(x) 
in a manner described above is assumed, its pixel value (new pixel value) C{x) can be represented by the following 
equation (1 0) from the above-mentioned equation (1 ): 



C(x) « (a(x)F(x) + (l-a(x))B(x))-G(x) (10) 
When the equation (10) is differentiated by x. the following equation (1 1) is provided. 

C* = ((F.B)a' + (F-Bla+ByQ m 
» ((F-B)a7-Q + ((F^Oy-lB-Gjla + (B*G)' ^ ^ 

Here, the texture of the picture obtained as the result of the feet that the pixel value of the foreground F and the pixel 
value of the background B are subjected to filtering by the low-pass filter R can be also considered to be both uniform 
similarly to the case shown in FIG. 4 when they are locally viewed. 

Accordingly, from a local point of view, the pixel values F*G and B*G which have been caused to undergo filtering 
can be considered to be both constant. As a result, these differential values {F*G)' and {B*G)' can be considered to be 
both zero (0). From this fact the equation (1 1) can be rewritten as indicated by the fblkjwing equation (12). 

C'.((F.B)arQ (12) 

Moreover, when locally viewed, as explained with reference to FIG. 4, the pixel values of the foregrouxl F and the 
background 8 can be considered to be both constant Accordingly, the equation (12) can be rewritten as indicated by 
the following equation (13):. 

C--(F.B)(o'QV (13, 

It 18 seen from the above equation (13) that change C of the pixel value between pixels of the picture C and the 
differential value (a'QV of the key signal for redudng alias have the proportional relationship in which difference F-B 
between the pixel value of the fbreground F and the pixel value of the background B is caused to be proportional con- 
stant 

[4] Case where the fbreground F or the background B is moving and pixels have d^inite size 

In this case, from the aboveKlescribed equations (1). (7) and (10). the pixel value C can be expressed by the fbl-' 
lowing equation (14): 

45 

C » ((a*Rf)(F*Rff) + (1-a*R0(B*Rf))*Q (14) 

When the equation (14) is differentiated by x. the fbltowing equatton (15) is provkjed: 

0' « ((F*Rf.B*Rb)(a-Rf) + (F*Rf.B"Rb)(a^Rf)+B-Rb)*G (1 5) 

Here, the texture of the picture obtained by filtering the pixel value of the fbreground F by the m& Rf to further allow 
It to undergo filtering by the low-pass filter G can be considered to be also uniform similarly to the case in FIG. 4 when 
locally viewed This similarly applies to the pixel value of the background B. 

Accordingly, from a local point of view, the pixel values F*RrG and B*Rb*G which have been caused to undergo 
filtering can be considered to be constant. As a result, these differential values (F*Rf *G)'. (B*Rb*G)' can be considered 
to be both zero (0). Further, when locally viewed, as explained with reference to FIG. 4, the pixel value of the fbreground 
F and the pixel value of the background B can be conskJered to be both constant From this fact the equation (15) can 
be rewritten as indicated by the fbltowing equation (16): 
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C » (F-Rf - B-Rb)(a'RfG)' (16) 

It is seen from the above equation (16) that the change C of the pixel value between pixels of the pcture C and the 
differential value (a'Rf 'G)' of the key signal such that the effect of motion blur is taken into consideration and that alias 
IS permitted to be reduced have proportional relationship in which difference F-Rf-B-Rb between the pixel value P-Rf 
and the pixel value B*Rb of the background in which the effect of motion blur is taken into consideration is caused to be 
the proportional constant. 

It is seen from the above-mentioned equations (3). (9). (13) and (16) that, in all of the above^esaibed cases of [11 
to [4]. the change C of the pixel value between pixels of the picture C and the differential value a' of the key signal have 
the proportional relationship, and the proportional constant is difference F-B between the pixel value of tfie foreground 
and the pixel value of the background constituting the picture C. Namely, even if motion blur or alias does not take place 
in the picture C. the proportional relationship between C and a* holds. Moreover, in the case where motion blur or alias 
takes place in the picture C. the proportional relationship between C* and a' holds also with respect to the relationship 
between the key signal a in which the influence of that motion blur is taken into consideration, or the key signal for per- 
mitting alias to be reduced and the pixel value C. In any case, the proportional constant becomes difference F-B 
between the pixel value of the foreground and the pixel value of the background constituting the picture C. 

Accordingly, a key signal a adapted to allow the proportional relationship to hold is such that in the case where 
motion blur takes place in the picture C. it becomes a key signal in which the influence of that motion blur is taken into 
consideration, and that in the case where alias takes place in the picture C. it becomes a key signal for altowing that 
20 alias to be reduced. This inventton is based on the abov&<jescribed theoretical ground (principle). 

FIG. 10 shows the configuration of an embodiment of the key signal generating apparatus adapted for generating 
a key signal on the basis of the above-mentioned principle. A picture C for generating key signal is inputted to a change 
detecting section 10. Then, the change detecting section 10 calculates (detects) change C (change information) of 
pixel value between pixels of the picture C to deliver this change C to a feature point detecting section 20 and an inte- 
rs gral value calculating section 40. 

The feature point detecting section 20 detects closed cun/e for path determination and feature point whk:h will be 
deserved later on the basis of the change C of the pixel value delivered from the change detecting section 1 0 to deliver 
the path determination ctosed curve and the feature point to an F-B detecting section 30 and an integral value calculat- 
ing section 40. 

30 The F-B detecting section 30 is supplied with the change C of the pixel value from the change detecting section 
10. and the path determination closed curve and tfie feature point from the feature point detecting section 20 and is 
supplied with the same picture C as the picture inputted to the change detecting section 10. Thus, the F-B detecting 
section 30 calculates (detects) difference F-B (difference information) between the pixel value of the foreground F and 
the pixel value of the background B of the pfcture C to deliver this difference F-B to the integral value calculating section 

35 40. 

The integral value catenating section 40 calculates the skeleton portion which will be desabed later of key signals 
a corresponding to the entirety of the picture C on the basis of the change C of the pixel value delivered from the 
change detecting section 1 0. tiie patii detemiination closed curve and the feature point delivered from the feature point 
detecting section 20. and the cSfference F-B delivered from the F-B detecting section 30 to deliver this skeleton portion 

40 to a key signal generating sectk)n 50. 

The key signal generating section 50 extracts the foreground F firom the ptoture C by using the skeleton portion 
delivered from the integral value calculating section 40. and to generate a key signal for synthesizing the extracted fore- 
ground F with any other picture. Namely, the key signal that the integral value cateulating section 40 outputs is only the 
skeleton portion of the key signals a corresponding to the entirety of the picture C. The key signal generating section 50 

4S interpolates portions except for the skeleton portion of the key signal to generate the key signal a corresponding to the 
entirety of the picture C. 

More practical operation of the key signal generating apparatus will now be described with reference to the flow- 
chart of Fia 11 . It is to be noted that picture C is assumed to be pteture on the xy plane, i.a, twoKfimensional pcture 
in tiie following descriptkxi. 

so At step Si . when picture C for g$ierating a key signal is inputted, that picture C is delivered to the change detecting 
section 10 and the F-B detecting section 30. Then, the change detecting section 10 is operative so that when picture C 
is inputted thereto, it calculates change C of the pixel value between pixels of that picture C. Namely, at this step Si. 
with respect to respective pixels constituting the picture C. e.g. . distance on the color space between pixels adjacent to 
each other is calculated. The distance thus cateulated is outputted as change (color change in this case) C of the pixel 

55 value between pixels of the picture C. 

A more practical example of the configuration of the change detecting section 10 of FIG. 10 is shown in FIG. 12. 
The picture C is delivered to an RGB decomposing section 1 1. The RGB decomposing section 1 1 decomposes the pic- 
ture C inta e.g., components of respective axes Erections in RGB (Red. Green. Blue) space. I.e.. R-component G-com- 
ponent and B-conriponent (hereinafter referred to as R. G. B components as the occasion may demand). These R. G. 
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B corrponents are respectively delivered to first (1 -st) order differential value calculating sections 12R. 12G and 1 2B. 

The first order differential value calculating sections 1 2R. 1 2G. 1 2B calculates respective first order differential val- 
ues R'. G'. B' to calculate directions and magnitudes (vectors) R\ G\ & of changes of respective components (i.e gra- 
dients of the curve assumed to be constituted by pixel values (respective components) of the picture C on the space in 

5 which axis indicating pixel values vertical to the xy-plane where the picture C exists is added to the xy^jlane. Then, vec- 
tors R\ G'. B' of respective changes of the R. G. B components are delivered to a vector synthesis section 13. 

The vector synthesis section 1 3 synthesizes the vectors R' G\ B* delivered from the first ader differential value cal- 
culating sections 12R. 12G. 12B. i.e.. synthesizes directions of respective changes of R. G. B components, and synthe- 
sizes magnitudes of the respective components to calculate change (color change) C of pixel values between pixels of 

w the picture C. 

The operation of the change detecting section 10 will now be described with reference to the flowchart of FIG. 13. 

At step S1-1 , the RGB decomposing section 1 1 decomposes the picture C into components of respective axes 
directions of R G B space, i.e.. R. G. B components. It is to be noted that the picture C may be decomposed into Y. U 
V components in color space except for the RGB space. e.g,. YUV space. Namely, the color space employed is not par' 
15 ticularly linked. In acWrtion to decomposing the picture C into components of respective axes directions of the color 
space, luminance components may be extracted therefrom to can-y out processing of the succeeding stage only with 
respect to the luminance components. Then, the R. Q, B components obtained at the step SM are respectively deliv- 
ered from the RGB decomposing section 1 1 to the first order differential value calculating sections 12R. 12G. 12B. 

At step S1-2. the first order differential value calculating sections 12R. 12G. 12B is operative so that when the R. 
20 G. B components are delivered thereto, they calculate the first order differential values R*. G'. B' of these RGB com- 
ponents. 

It is to be noted that in this embodiment first order differential values of respective conponents are determined by 
carrying out. e.g., convolution of respective components and sobel operator, etc. Namely, at the first order differential 
value calculating sections 12R. 12G. 12B. values obtained by carrying out convolution of x or y direction components 

25 Of the respective R. G. B conponents and sobel operator SO^ or SOy shown in FKjL 14 are caused to be respectively 
first order differential values fx. fy in the x or y direction. 

Rrst Ofder differential values of the R, Q. B conponents detenrtned in a manner described above, i.e.. changes of 
the R, Q, B components are such that tfieir magnitudes (gradients) are maximum at respective pixels. Accordingly, the 
directions of changes of respective components are in con-espondence with the directions In which the edge intensity 

30 is maxin^. Namely, at this step SI -2. as shown in FIG. 15. changes of the R. Q. B conponents with respect to the 
direction caused to be in correspondence with the direction in which the edge intensity is maximum of any (arbitrary) 
directions of 360 degrees with respect to the remarked pixel (the portion to ¥irhich slanting Unas are attached in FIG. 15) 
are determined. 

Then, first order differential values R\ Q\ B' of R, G. B components respectively detemriined at the first order differ- 
35 ential value calculating sections 12R. 12G, 12B (these values are vectors because they have magnitude and direction 
as described above) are delivered to the vector synthesis section 13. 

At step Si -3. the vector synthesis section 1 3 synthesizes first order differential values R'. G", B* to calculate change 
(color change) C of the pixel value between pixels of the picture C. Namely, if first order differential values R*. G\ B* are 
respectively represented by (r,. ry). (g^, gy). (b,. by), and C is represented by (c,. Cy) (a and b of (a, b) respectively indi- 
go cate x-component and y-component). At this step SI -3. x component Cx and y component Cy of change C* of the pixel 
value are respectively determined by the expression r^^^ and ry+gy+toy Then, change C* of the pixel value deter- 
mined in a manner as stated above at the change detecting section 1 0 is delivered to the feature point detecting section 
20 and the integral value calculating section 40. Thus, the processing at the change detecting section 1 0 is conpleted. 
At step S2 (FIG. 11 ). the feature point detecting section 20 is operative so that when the change C of the pixel value 
45 is delivered thereto from the change detecting section 1 0, it detects the path detennination dosed curve and the feature 
point from the change C of that pixel vaiua 

A more practical example of the configuration of the feature point detecting section 20 is shown in FIG. 16. A 
boundary (border) detecting section 21 is supplied with the change C of the pixel value from the change detecting sec- 
tion 1 0. Thus, the boundary detecting section 21 detects, from the change C* of the pixel value, boundary between the 
50 foreground (object) and the background 6 whfeh constitute the picture C Namely, the boundary detecting section 21 is 
operative, as shown in FIG. 17A. for example, to detect the points (pixel) where the change C of the pixel value is well 
to detect as the boundary between the foreground F and the background B (boundary of the object in the picture C). 
closed curve constituted by connecting such points (since thte dosed cun/e is used for determining integral path which 
will be desaibed later, it will be called hereinafter path detennination dosed curve as the occasion may demand). Then. 
55 the path determination dosed curve detected at the boundary detecting section 21 is delivered to ttie feature point 
determining section 22. 

The feature point determining section 22 determines, as the feature point predetemnined points on the path deter- 
minatton ck)sed cunre delivered from ttie boundary detecting section 21 as shown in FIG. 17B. for example. Namely, 
the feature point detennining sectk)n 22 detects. e.g.. the points where curvature is high of points constituting the path 
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determination closed curve to determine those points as the feature point 

Moreover, the feature point determining section 22 is operative, as shown in FIG. 17C. for exanple to determine 
the closed curve sun-ounding the path determination closed curve (the portion indicated by dotted lines' in FIG 1 7C) 
and divides the area of the background B surrounded by that closed curve and the path determination closed curve into 
several areas where the change C of that pixel value is relatively constant (hereinafter referred to as background con- 
stant area as the occasion may demand). Further, the feature point determining section 22 partitions the path determi- 
nation closed curve in the background constant area units so that divided very small areas are provided. In addition the 
feature point determining section 22 detects middle points in respective very small areas to determine those pointe as 
the feature point 

The operation of the feature point detecting section 20 will now be described with reference to the flowchart of FIG. 

18. 

At step S2-1. the boundary detecting section 21 is operative so that when the change C of the pixel value is deliv- 
ered thereto from the change detecting section 10. it detects the points where the change C* of that pixel value is great 
to connect those points to thereby detect a path determination closed cun^a Then, the path determination closed curve 
IS is delivered to the feature point determining section 22. 

At step S2-2. the feature point determining section 22 is operative so that when the path determination closed curve 
IS delivered thereto from ttie boundary detecting section 21. it determines, as the feature point, predetermined points 
on the path determination closed curve. Then, the path determination closed curve and the feature points respectively 
obtained at the boundary detecting section 21 and the feature point determining section 22 are delivered to the F-B 
detecting section 30 and the integral value calculating section 40. Thus, the processing at this feature point detectina 
section 20 is completed. 

At step S3 (FIG. 1 1). the F-B detecting section 30 is operative so that when the path determination closed cun^e 
and the feature points are delivered ttiereto from ttie feature point detecting section 20. it calculates difference F-B 
between ttie pixel value of the foreground F and ttie pixel value of the background B which constitute the picture C. 

25 A more practical example of the configuration of the F-B detecting section 30 is shown in FIG. 19. The very small 
area detecting section 31 is supplied, in addition to the picture C (picture value), witti the feature points and the path 
determination closed cun/e from the feature point detecting section 20. Thus, the very small area detecting section 31 
detects the very small area including the feature points as shown in FIG. 20A, for exanple. Namely, the very small area 
detecting section 31 divides the patii detennination closed curve into very small line segments at points equi-distant 

30 from tine feature point as shown in FIG. 20B in ttie state where a portion of FIG. 20A is enlarged to detect, as the very 
small area (the portion to which slanting lines are attached in FIG. 20B). the range spaced by a predetemiined distance 
(e.g.. distance corresponding to four pixels, etc.) tomrd the skle of the foreground F or the background B from that line 
segment Moreover, the very small area detecting section 31 extracts, with respective to respective very small areas, 
pixels constituting the foreground F included within those areas (pixels existing at the foreground F side from the path 

35 determination closed cun^e. I.e.. the portion to which slanting lines going down to the left are attached in FIG. 20B) to 
deliver pixel value of those pixels (hereinafter referred to as F boundary pixel value as the occasion may demand) to a 
color distribution forming circuit 32F. and extracts pixels constituting the background B (pixels existing at the background 
B skje from tiie path determination ck)sed curve. i.e.. ttie portion to whwh slanting lines going down to the right are 
attached in FIG. 20B) to deliver pixel value of those pixels (hereinafter referred to as 6 boundary pixel value as ttie occa- 

40 sion may demand) to a cokx distribution forming section 32B. 

The color distribution forming sections 32F. 32B respectively determines cotor distributions on the color space, e.g., 
RGB space, etc. of the F boundary pixel values and the B boundary pixel values e/ery very small areas delivered from 
the very small area detecting section 31 to respectively deliver them to center of gravity detecting sections 33F. 33B, 
The center of gravity detecting sections 33F, 33B respectively determine centers of gravity of the color distributions of 

45 the F boundary pixel values and ttie B boundary pixel values delivered from ttie cotor distrtoution forming sections 32F. 
32B to considers respective centers of gravity as pixel values of the foreground F and ttie background B of pixels in ttie 
gray area to deliver ttiem to a difference catoulating section 34. 

The difference calculating section 34 is operative, as shown in FIG, 21 . for example, to subtract, from tiie center of 
gravity of ttie cola distribution of the F boundary pixel values delivered from the center of gravity detecting section 33F. 

so the center of gravity of the cotor distribution of ttie B boundary pixel values delivered from ttie center of gravity detecting 
section 338 to output ttie subtraction result as difference F-B between ttie foreground F and ttie background B (different 
of pixel value between ttie pixel value of the foreground F and pixel value of the background B. and ttiis difference is 
thus considered to be vector similarty to C) in ttie very small area (or at feature point included wittiln that very small 
area). 

55 The operation of ttie F-B detecting section 30 win now be described witti reference to ttie f towchart of FIG. 22. 

At step S3-1 . ttie very small area detecting section 31 detects ttie very sman area, which has been explained witti 
reference to FIG. 20. on ttie basis of ttie feature points and ttie patti determination closed cun^e delivered from ttie fea- 
ture point detecting section 20 to further extract F boundary pixel values and B boundary pixel values in that very small 
area to respectively deliver ttie F boundary pixel value and ttie B boundary pixel value to the color distribution forming 
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sections 32F. 32B. 

At step S3-2. the color distribution forming sections 32F. 32B respectively determines color distributions on the 
RGB space of the F boundary pixel values and the B boundary pixel values in a certain very small area to deliver them 
to the center of gravity detecting sections 33F. 33B. 

At step S3-3. the center of gravity detecting sections 33F. 338 respectively calculates (detect) centers of gravrty of 
color distributions of the F boundary pixel values and the 8 boundary pixel values to deliver them to the difference cal- 
culating section 34. 

At step S3-4. the difference calculating section 34 determines difference value between the center of gravity of the 
color distribution of the F boundary pxel values delivered from the center of gravity detecting section 33F and the center 
of gravity of the color distribution of the B boundary pixel values delivered from the center of gravity detecting section 
33B to output this difference value as difference F-B between the fbreground F and the background 8 at the feature 
point included within the very small area. 

At step S3-5. after the difference F-B between the foreground F and the background at the feature point included 
within a certain very small area is obtained in a manner as stated above, whether or not the difference F-B is deter- 
15 mined with respect to all very small areas detected at the very small area detecting section 31 (all feature points 
detected at the feature point detecting section 20) is judged. In the case where ft Is judged at the step S3-5 that the dif- 
ference F-B has not yet been determined with respect to all very small areas, the processing operation returns to the 
step S3-2 to carry out the processing at step S3-2 and steps subsequent thereto with respect to the very small areas 
where the difference F-B has not been detenroned. On the other hand, in the case where it is judged at the step S3-5 
that the difference has been determined with respect to all the very small areas, the processing at the F-B detecting 
section 30 is completed. Then, the differences F-B which have been determined with respect to all the very small areas 
are delivered to the integral value calculating section 40. 

It is to be noted that the direction of ttie difference F-B (vector) between the fbreground F and the background B 
obtained in a manner as described above Is In con^espondence with (sub^antially in correspondence with) the direction 
25 in which the edge Intensrty becomes maximum, and the direction of F-B and the direction of C are in correspondence 
with each other (substantially in correspondence with each other). 

Further, while the F-B detecting section 30 shown in FIG. 19 determines the very small area on the basis of tfie 
path determination closed curve (tiie boundary between the foreground F and the background B) obtained from the 
change C of the pixel value to determine the difference F-B from the very small area, the difference F-B may be deter- 
30 mined. e.g.. in a manner stated below at the F-B detecting section 30. In more practical sense, e.g.. not only the frame 
of the picture C but also frames constituting moving pkAjre before and after that frame is iiputted to the F-B detecting 
section 30 in a time series manner. Then, the F-B detecting sectfon 30 stores the frame inputted at present until the 
subsequent frame is inputted. In the case where picture C for generating a key signal is injHJtted. frame of that picture 
C and the stored frame preceding (earlier) by one are caused to undergo the so-called block matching to detect the 
35 boundary between the foreground F and the background B on the basis of tiie above-mentioned result and the result of 
the btock matching preceding thereto as occasion demands. Then, ttie F-B detecting section 30 detemiines feature 
points on the patii determination ctosed cun^e estimated from the boundary by using the boundary to detect the very 
small area including respective feature points similarly to the case at the steps S3-1. S3-2 of FIG. 22 to recognize 
respective distrtoutlons of the fbreground F and the background B which constitute the pkrture C every respective very 
small areas to determine centers of gravity of respective cotor distributions of the foreground F and the background 8 
similarly to the case at the steps S3-3. S3-4 of FIG. 22 at times subsequent thereto to further determine direction of the 
position of the center of gravity of tfie color distrtoution of the background F with respect to the position of the center of 
gravity of the color distribution of the background B. and cfistance on the color space tfierebetween. i.e.. the cfifference 
F-B between the fore is unnecessary to deliver the path determination closed curve and the feature points from the fea- 
45 ture point detecting section 20 to the F-B detecting section 30. It is to be noted that the detail of ttie method of detecting 
the boundary between the fbreground F and the background 8 by using block matching as described above is dis- 
closed, e.g., in the Japanese Patent Application Na 1 64792/1 995, etc. 

fwloreover, whUe the method of detecting the boundary between the faeground F and the background B is not lim- 
fted to the method using change O of the pixel value and the method using block matching as desaibed above, detec- 
so tion of the boundary between the foreground F and the background 8 may be carried out by instruction from users as 
described later in FIG. 30. for example Further, the boundary between the foreground F and the background 8 may be 
detected by employing in combination, e.g., ttie mettrod by instruction from users and the method using change C* of 
pixel value. Furthermore, the boundary between the foreground F and ttie background B may be detected by enploying 
in combination, e.g.. the mettKXj by instruction from users and ttie mettiod using bk)ck matching. Namely, eg., an 
55 approach may be enployed to detect ttie boundary between ttie fbreground F and ttie background B by instruction from 
users witti respect to ttie first frame, and to detect the boundary between ttie foreground F and ttie background B by 
block matching witti respect to frames subsequent thereto. 

'^wrtw^fi '"ettiod of detecting ttie difference F-B is not limrted to ttie above-descrft)ed mettiod. but may enploy 
other methods. 
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Moreover, while after the processing of the step S2. th processing of the step S3 is carried out in the flowchart 
shown in FIG. 11 . in the case where there is no necessity of delivering the path determination closed curve and the fea- 
ture points with respect to the F-B detecting section 30 from the feature point detecting section 20. the processing of 
the step S2 and the step S3 may be carried out in an order opposite to the above, or may be carried out at the same 
time (in parallel). The difference F-B determined in a manner as described above at the step S3 (FIG. 1 1) is delivered 
from the F-B detecting section 30 1 the integral value calculating section 40. 

At step S4. the integral value calculating section 40 normalizes change C of pixel value delivered from the change 
detecting section 10 by absolute value F-Bl of difference F-B delivered from the F-B detecting section 30 

At step S5. the integral value calculating section 40 integrates the normalized value C'/jF-BI along e g the direc- 
tK)n in which change C* of pixel value is calculated (detected) to thereby calculate the skeleton portion of the key signal 

In more practical sense, since the change C of pixel value between pixels of the pteture C and the differential value 
a' of the key signal have the proportional relationship therebetween as described above, and the proportional constant 
IS difference F-B between the foreground and the background which constitute the picture C. i.e.. the expression 
C a(F-B) a' holds, the key signal a(p) can be determined by the following equation (17). 

'**^''|F(36(p)|* (17) 



In the above equation. a(p) means a key signal with respect to pixel at the position (point) p(x. y) constituting the 
picture C. and F(p) and B(p) respectively means pixel values of pixels at the position p(x. y) of the foreground F and the 
background B which constitute the picture C. Further. C(p) means pixel value of pixel at the position p(x. y) constituting 
the picture C. It is to be noted tfiat description of ttie argument (p) of the function will be hereinafter omitted as the occa- 
sion may demand. 

Accordingly, at the above-described step 84. C'/F-b: constituting ttie right side of the equation (1 7). i.e., normalized 
value obtained by normalizing the change C of the pixel value by the absolute value JF-b: of difference is determined. 
At this step S5. integration at the right side of the equation (17). i.e.. integration of the normalized value C7|F-B| along. 
e.g., the direction in which change C of the pixel value is detected is canied out V^ue obtained as the result of inte^ 
grating value obtained by normalizing change C of the pixel value by ttie absolute value F-B of difference witiwut inte- 
grating ttie change C of ttie pixel value as rt is. i.e.. ttie key signal a takes value wittiin ttie range from 0 to 1 . 

The integration canried out at the step S5 will now be desaibed in more detail. When space in which ttie z-axis per- 
pendicular to ttie xy plane where ttie picture C exists is provided is assumed to be on the xy-plane is assumed as shown 
in FIG. 23. for example, to take pixel value in the z-axts direction, plane (curved surface) V consisting of pxel values is 
formed wittiin ttiat space. It shouU be noted that if ttie picture C is cotor picture (pfcture image), ttiere are ttiree compo- 
nents such as RGB or YUV. etc. as descrbed above, but the pixel value is assumed to consist of one conponent for 
the brevity of explanation. 

In this case, gradient at an intersecting point of a straight line passing ttirough a certain point p on the picture C and 
perpendicular to ttie xy-plane and the plane V, i.e.. change of pixel value C(p) of pixel located at ttie position p can be 
represented as vecta in ttie maximum inclination direction when ttie plane V is viewed from ttiat point C(p) (vector 
obtained by calculating gradient of the plane V at the pant C(p)). 

Here, ttie maximum inclination direction is ttie directwn In which in ttie case where a ball having size assumed to 
be infinitely small is placed at ttie point C(p). that ball rolls. 

Let now suppose that in ttie case where a ball is placed at ttie point C(p) and ttiat ball falls (rolls) down to ttie point 
C(po). ttie locus (patti) ttiat ttie ball depk:ts on ttie plane V is U. In ttiis instance, ttie integration of the equation (17) 
means ttie inteyation along ttie path (hereinafter refen-ed to as ttie maximum inclination direction path as the occasion 
may demand), and ttie equation (17) means ttiat ttie integration expressed ttiereby is candied out, whereby key signal 
a(p) with respect to the pixel located at the position p can be determined. 

Accordingly, if the key signal a(po) witti respect to ttie pixel located at ttie position po can be recognized, it is pos- 
sible to determine ttie key signal a(p) witti respect to ttie pixel located at ttie positton p in accordance witti ttie following 
equation (18) from the equation (17): 



In ttie above equation, integral symbol in which U is attached at ttie iwer right means ttie integration along ttie 
maximum inclination direction patti U. 

At step 35. e.g., cateulation of ttie right side of the equation (18) is can-ied out. or the like, whereby the key signal 
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a (the skeleton portion of the key signal) is determined. 

It is to be noted that it is not necessarily required to carry out the integration of the right side of the equation (18) 
along the maximum inclination direction path U. Namely, integration along path (FIG. 23) which is not the maximum 
inclmation direction path U as the path from the point C(p) up to the point C(po) may be carried out. Namely, since the 
5 integration of the nght side of th equation (1 8) means the integration of gradients at respective points on the maximum 
inclination direction path U. in the case wher integration is carried out along other path . the component in the direc- 
tion of the maximum inclination direction path U of gradients at respective points on the path is integrated (Accord- 
ingly, this integration can be also conskJered to be eventually the same as the integration along the maximum inclination 
direction path U), thereby making it possible to determine the same key signal a as that in the case where integratton 
10 is carried out along the maximum inclination direction path U. 

It is to be noted that since in the case where integration is earned out along paths except for the maximum inclina- 
tion direction path U. the quantity of operations is increased to more degree as compared to the case where integration 
IS camed out along the maximum inclination direction path, it is preferabfe to carry out integration along the maximum 
inclination direction path U as far as poss3)le. 

An example of the configuration of the integral value calculating section 40 of FIG. 10 for canTing out a processing 
as desaibed above is shown in FIG. 24. The path determination closed cun/e and the feature points from the feature 
point detecting section 20 are delivered to an integral path determining section 41 . and change C of the pixel value from 
the change detecting section 10 and difference F-B from the F-B detecting section 30 are delivered to a normalization 
executing section 42. 

'20 The integral path determining section 41 determines integral path for carrying out integration of the equation (1 8) 
on the basis of the path detennination dosed curve and the feature points delivered from the feature point detecting 
section 20. Namely, as shown in FIG. 25A. the integral path determining section 41 determines, as the integral patti with 
respect to corresponding feature point a line in a normal direction of the path deterntination ctosed curve and passing 
through the corresponding feature point. The integral path with respect to the feature point determined in a manner as 

2S descnbed above is delivered to the normalization executing section 42 and the integration executing section 43. It is to 
be noted that the integral patti detemiined in this way is the patti substantially along the maximum indfnation direction 
patti U. 

The normalization executing section 42 determines nomnalized value C/1F-B| (normalized information) obtained by 
normalizing change C of pixel value of pixel existing on the integral path delivered from the integral path determining 
30 section 41 by absolute value |F-B| of difference F-B to deliver it to the integration executing section 43. The integration 
executing section 43 canies out integration of the normalized value C'/|F-B| along the integral path from the integral 
path determining section 41 in accordance with the equatton (18) to determine key signal a on the integral path. The 
key signal on the integral path is delivered to the closed curve generating section 44. 

The closed curve generating section 44 generates a PI closed cunre and a P2 closed cunre whfoh wilt be described 
35 later from tiie key signal a delivered from tiie integration executing section 43. 

fMamely. while the integratfon In accordance with the equation (18) is carried out along the integral path shown in 
FIG. 25A at the integration executing section 43. whereby the key signal a on the integral patti can be obtained, ttiat key 
signal a becomes equal to 0 (zero) at ttie boundary between ttie gray area and the background B, and becomes equal 
to 1 at the boundary between ttie gray area and the foreground F Accordingly, as shown in FIG. 25A, the dosed curve 
generating section 44 determines points where key signals a on respective integral paths become equal to 0 (points 
where a is equal to 0. which will be referred to as pi as ttie occasion may demand hereinafter), and determines points 
where key signals a become equal to 1 (points where a becomes equal to 1 . which will be referred to as P2 as the occa- 
sion may demand hereinafter), thus to connect the points Pi on respective integral paths to each other, and to connect 
the points P2 on the respective integral path to each ottier. Then, as shown in FIG. 25B, the dosed curve generating 
45 section 44 outputs, as PI dosed curve, the dosed cun^e obtained by connecting ttie points p^ to each ottier. and out- 
puts, as P2 dosed cun^e. ttie dosed curve obtained by connecting the points P2 to each ottier. The key signal a 
obtained at ttie integration executing section 43 and ttie P1 dosed cun^e and ttie P2 closed cun/e obtained at ttie dosed 
cun^e generating section 44 are delivered to ttie key signal generating section 50 (FIG. 10). 

The operation of ttie integral value cateulating section 40 will now be desaibed witti reference to ttie ftowchart of 
50 FIG. 26. 

At step S4-1. ttie integral patti determining sectfon 41 allows any one of featjre points delivered from ttie feature 
point detecting section 20 to be remarked feature point to determine a straight line passing ttirough the remarked fea- 
ture point and in nonnal direction of the patti determination dosed cun/e delivered from feature point detecting section 
20 as ttie integral patti witti respect to ttie remarked feature point The integral patti witti respect to ttie remarked feature 
55 point is delivered to ttie normalization executing section 42 and ttie integ-ation executing section 43. 

At step S4-2. ttie normalization executing section 42 divides the change C of ttie pixel value of ttie pixel existing on 
ttie integral patti delivered from ttie integral patti determining section 41 by absolute value IF-B,' of ttie difference which 
is ttie difference F-B delivered from the F-B detecting section 30 and is ttie difference detennined from ttie very small 
area (FIG. 20) including ttie remari«ed feature point to deliver ttie normalized value C/|F-B| obtained as ttie result 
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thereof to the integration executing section 43. 

At step 84-3. the integration executing section 43 integrates the nonnalized value C'/if-b: in accordance with the 
equation (18) along the integral path from the integral path determining section 41 by using the integral path and the 
normalized value C'/IF-B| with respect to the pixel existing on the integral path respectively delivered from the integral 

5 path determining path and the normalization executing section 42 to calculate a key signal a on that integral path This 
key signal a is delivered to the closed curve generating section 44. In this case, at the integration executing section 43 
e.g.. the remarked feature point is regarded as the point where a becomes equal to 0.5. and the integration of the equa- 
tion (18) IS carried out with the remarked feature point being as the initial point Accordingly, in this example, the inte- 
gration of the equation (18) is carried out in the state where a(po) is caused to be 0.5. 

10 At step S4-4. the integral path determining section 41 judges whether or not all of feature points delivered from the 
feature pant detecting section 20 have been caused to be the remarked feature point. In the case where the integral 
path determining section 41 judges that all of the feature points from the feature point detecting section 20 have not yet 
been caused to be the remarked feature point, its processing operation returns to the step S4-1 . As a result, at tiie inte- 
gral path determining section 41 . feature point which is not yet caused to be the remarked feature point is newly caused 

15 to be the remarked feature point. At times subsequent ttiereto. processing as described above will be repeated. 

On the other hand, in ttie case where the integral path determining section 41 has judged at the step S4-4 that all ' 
the feature points have been already to be caused to be ttie remarked feature point, i.e.. in the case where the integra- 
tion of the equation (18) atong all of integral paths passing ttirough respective feature points delivered from the feature 
point detecting section 20 is carried out whereby key signals a on those integral paths (hereinafter referred to as the 

20 skeleton portton as the occasion may demand), the processing proceeds to step S4-5. 

At the step S4-5, the dosed curve generating section 44 detects ttie point p, (ttie point of a = 0) and ttie point pg 
(tfie point of a o 1) from ttie key signals a on respective integral paths delivered from ttie integration executing section 
43 to connect ttie points pi to each other and to connect ttie point P2 to each ottier to ttiereby generate a P1 ck>sed 
curve and a P2 closed curva Thus, ttie processing at ttie integral value calculating section 40 is conpleted. 

25 It is to be noted ttiat at ttie integration executing section 43. the integration ak)ng ttie integral path delivered from 
ttie integral patti determining section 41 is carried out witti respect to. e.g.. segment (line segment) (hereinafter referred 
to as path segment as ttie occasion nray demand) between ttie point where ttie key signal a becomes equal to 0 (ttie 
point pi) and ttie point where it becomes equal to 1 (the point pa) of that integral patti. i.e.. ttie integral patti within ttie 
gray area. This is because since key signals a in areas except for the gray area take any one of values 0 and 1. it is 

30 unnecessary to determine a with respect to such areas in accordance witti ttie equation (18). Accordingly, only the key 
signal a on ttie path segment is calculated at ttie integration executing section 43, and ttie abov&described skeleton 
portion means ttie patti segments in all integral pattis delivered from the integral patti determining section 41 . 

The skeleton portion of ttie key signal, and the Pi closed curve and ttie P2 dosed cun^e obtained at ttie Integral 
value calculating section 40ln a manner as descried above are delivered to the key signal generating section 50. 

35 At Step S6 (Fia 1 1 ). ttie key signal generating section 50 carries out interpolation witti respect to ttie skeleton por- 
tion of ttie key signal to generate a key signal a con-esponding to ttie entirety of ttie pictwe C. Thus, ttie processing at 
the key signal generating section 50 is completed. 

A more practical example of the configuration of ttie key signal generating section 50 is shown in FIG, 27. An inter- 
polating section 51 for inside of ttie gray area is supplied witti ttie skeleton portion of ttie key signal from the integral 

40 value calculating section 40 and key signals on ttie Pi dosed curve and ttie P2 dosed curve from 0/1 allocating section 
52 wrtch will be described later. Thus, ttie interpolating section 51 for inskJe of ttie gray area carries out interpolation of 
the key signal in ttie area except for ttie skeleton portion, ttie Pi dosed curve and ttie P2 closed curve of ttie gray area 
shown in FIG. 28A in such a manner that slanting tines are attached (ttie area sun-ounded by ttie skeleton portion, ttie 
PI dosed curve and ttie P2 dosed cun^e). 

45 Namely. Fia 28A shows xya space in which picture C is disposed on ttie xy-plane and axis (a-axis) perpendicular 
to ttiat xy-plane is taken. It is now assumed ttiat key signals at respective points on ttie picture C are plotted in ttie a- 
axis direction wittiin ttiis xya spaca Since ttie key signals delivered to the interpolating section 51 for ttie inside of ttie 
gray area are the skeleton portion of the key signal (key signal on ttie patti segment), key signals on the PI closed curve 
(since ttie PI dosed curve is ttie curve in which points Pi where a is equal to 0 are connected, values of key signals 

50 ttiereat are 0 (zero)), and key signals on ttie P2 dosed cun/e (since ttie P2 dosed curve is ttie curve in which points pg 
where a is equal to 1 are connected, values of key signals ttiereat are 1). points con-esponding to ttie skeleton portion 
of ttie key signal are plotted on ttie patti segment, and points of a a 0 and a o 1 are respectively plotted on ttie Pi dosed 
cun^e and ttie P2 closed curve. Thus, a frame as indicated by ttiick line in FIG. 28A (hereinafter refeaed to as ttie skel- 
eton frame as ttie occasion may demand) is formed on ttie xya spaca 

55 Then, ttie interpolating section 51 for the inside of ttie gray area generates curved surface Q for interpolating ttie 
inside of ttie skeleton frame to allow a that ttiis cun/ed surface Q provides (designates) to be key signals a at respective 
points wittiin ttie area formed when ttie cun^ed surface Q is projected onto the xy-plane (portions to which slanting lines 
are attached in FIG. 28B which will be refeaed to as patch areas as ttie occasion may demand). 

In this case, ttie interpolating section 51 for ttie inside of ttie gray area carries out as shown in FIG, 288. interpo- 
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lation of key signals as described abov with respect to the areas surrounded by the P1 and P2 dosed curves and the 
path segment i.e.. alt of the patch areas. 

The skeleton portion of the key signal, the key signals {a = 0) on the PI closed curve, the key signals (a » 1) on the 
P2 closed curve, and th key signals in th patch area obtained by interpolation are delivered from the interpolating sec- 
tion 51 for the inskle of the gray area to a synthesis section 54. 

The 0/1 allocating section 52 is supplied with the Pi closed curve and the P2 dosed curve. The 0/1 allocating sec- 
lion 52 respectively allocates 0 and 1 as key signals a on the Pi dosed cun^e and the P2 dosed curve to deliver them 
to the interpolating section 51 for the inside of the gray area. Moreover, an interpolating section 53 for area except for 
the gray area is supplied with the PI dosed cun/e and the P2 closed curve. As shown in FIG. 28B. the interpolating 
section 53 for area except for the gray area interpolates 0 as key signals at respective points within the area put 
between the picture frame of the picture C and the PI dosed curve (the area considered to be completely bad^ound 
B. which will be referred to as aO area as the occasion may demand hereinafter), and interpolates 1 as key signals at 
respective points within the area sunrounded by the P2 dosed curve (the area considered to be completely the fore- 
ground f. which will be referred to as a1 area as the occasion may demand hereinafter), thus to deliver them to the syn- 
thesis section 54. 

The synthesis section 54 synthesizes (arranges (allocates) at correspondng positions within the picture frame of 
the picture C) the skel^on portion of the key signal, the key signals on the PI dosed cun^e. the key signals on the P2 
closed curve, and the key signals of the patch area obtained by interpolation from the interpolating section 51 for the 
inside of the gray area, and the key signals of the ceO area and the key signals of the al area from the interpolating sec- 
tion 53 for area except for the gray area, thereby to generate a key signal con^esponding to the entirety of the picture C. 

The operation of the key signal generating section 50 will now be described with reference to the flowchart of FIG 

29. 

At step S5-1. the 0/1 allocating section 52 respectively allocates 0. 1 to the PI dosed cun/e and the P2 dosed 
cun/e to generate key signals on the PI dosed curve and key signals on the P2 dosed cun/e. The key signals on the 
PI dosed cun/e and the key signals on the P2 dosed curve are delivered to the interpolating section 51 for the inskSe 
of the gray area. 

At step S5-2, the Interpolating section 51 for ttie inside of ttie gray area is operative so that when the key signals 
on the PI dosed curve and the key signals on the P2 dosed curve are delivered ttiereto from ttie 0/1 allocating section 
52. ft forms the skeleton frame as has been explained with reference to FIG. 28A to further generate curved surface 
(interpolation curved surface) Q for interpolating the inside of the skeleton portion. 

At step S5-3. the interpdating section 51 for the inskle of the gray area generates key signals of the patch areas 
on the basis of tiie curved surface Q to deliver them to the synthesis section 54. 

At step S5-4. ttie interpolating section 53 for area except for the gray area interpolates 0 as key signals at respec- 
tive points within ttie aO area, and interpolates 1 as key signals at respective points wittiin the al area, thus to deliver 
ttiem to ttie synttiesis section 54. 

At step S5-5. ttie synttiesis section 54 synthesizes ttie skeleton portion of ttie key signal, ttie key signals on the Pi 
closed curve, ttie key signals on ttie P2 dosed area and ttie key signals of ttie patch areas obtained by interpolation 
delivered from the interpolating section for ttie inskle of the gray area, and ttie key signals of ttie aO area and ttie key 
signals of ttie al area delivered from ttie interpolating section 53 for area except for ttie gray area to generate a key 
signal corresponding to ttie entirety of ttie pfcture C. Then, ttie synttiesis section 54 converts this key signal (continuous 
key signal) into key signals of pixel units by implementing, ag.. fitering, ete. ttiereto as prevfously described to output 
ttiese key signals as a key signal for extracting ttie foreground F from ttie pictiire. and allowing ttie extracted foreground 
F to undergo synttiesis (keying) witti respect any ottier picture. Thus, ttie processing at ttie key signal generating section 
50 (S completed. 

It is to be noted ttiat the processing of ttie step S5-4 may be carried out prior to the processing of the steps S5-1 to 
S5-3. or may be carried out at ttie same time (in parallel). 

Furttier. while ttie key signals of ttie patch areas are obtained from ttie curved surface Q for interpolating ttie skel- 
eton frame in ttie above-described case, ttie key signals of ttie patch areas may be determined, in addition to ttie above, 
e g., in a manner as described belm. Namely, an approach may be employed to determine key signals at points adja- 
cent to ttie patti segment, the PI dosed curve and ttie P2 cfosed cun/e within ttie gray area on the basis of ttie skeleton 
portion of the key signal, ttie key signals on ttie PI dosed curve and ttie key signals on ttie P2 dosed cun/e which have 
been already obtained to sequentially, at times subsequent ttiereto. key signals at points adjacent to points where ttie 
key signals have been determined on ttie basis of already determined key signals. 

Moreover, while such an approach is employed in ttie above-described case to determine key signals of ttie patch 
areas from ttie curved surface Q for interpolating the skeleton frame formed by pfotting points con-esponding to ttie skel- 
eton portion of ttie key signal onto ttie path segment and respectively plotting respective points of a = 0. a = 1 onto ttie 
PI dosed curve and ttie P2 dosed curve, ttie key signals of ttie patch areas may be determined, in addition to ttie 
above-mentioned approadi. e.g.. from a cun/ed surface for interpolating frame obtained by furttier plotting points of a 
= 0.5 as well onto ttie patti determination dosed curve. 
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While the path determination dosed curve, i.e.. boundary between the foreground F and the background B is deter- 
mined from change C of the pixel value in the embodiment of FIG. 10. the path determination closed cun/e may be 
determined on the basis of instruction from users of the apparatus. 

A more practical example of the configuration of a key signal generating apparatus for calculating a path determi- 
nation closed curve on the basis of instaxrtion from a user is shown in FIG. 30. It is to be noted that the same reference 
numerals are respectively attached to portions corresponding to the key signal generating apparatus shown in FIG. 10 
and the explanation will be omitted as the occasion may demand in the following description. 

Namely, at this key signal generating apparatus, a feature point detecting section 60 is provided in place of the fea- 
ture point detecting section 20. and an operation sectk)n 1 is newly provided. In addition, other portions of this key sig- 
nal generating apparatus are respectively constituted in the same manner as in the key signal generating aoparatus 
shown in FIG. 10. » « -rr-^ 

The operation section 1 is conrposed of. e.g.. tablet and pen or mouse, etc.. and is operated when a user desig- 
nates the boundary between the foreground F and the background B while looking at picture C displayed on display (not 
shown). In additk)n. an operation signal corresponding to that operation is delivered to ttie feature point detecting sec- 

IS tionSO. 

The feature point detecting section 60 detects the path determination closed curve and the feature points on ttie 
basis of \he operation signal from the operation section 1,. A more practical example of ttie configuration of ttie feature 
point detecting section 60 is shown in FIG. 31. It is to be noted that in ttie feature point detecting section 60 shown in 
FIG. 31, ttie same reference numerals are respectively attached witti respect to portions corresponding to ttie feature 
20 point detecting section 20 shown in FIG. 16 and ttieir explanation wilt be omitted as ttie occasion may demand in ttie 
following description. Namely, a line ttiinning section 61 is provided, in place of the boundary detecting section 21. at 
the feature point detecting section 60. and ottier portions of the feature point detecting section 60 are constituted in the 
same manner as in ttie feature point detecting section 20. 

The line ttiinning section 61 recognizes line designated by us^ (hereinafter refen'ed to as designation line as ttie 
occasion may demand), wrhfch corresponds to ttie boundary between ttie foreground F and ttie background B. on ttie 
basis of ttie operation signal delivered from ttie operation sectkDn 1 to carry out line ttiinning to ttiereby detect a patti 
determination closed curva For example, ttie designation line is displayed on a display where picture C is displayed in 
a manner overlakJ upon ttie picture C. and is line having wkfth of one pixel or more as shown in FIG. 32A. In view of 
this, ttie line ttiinning section 61 determines, as shown in FIG. 32B. for example, in ader to determine patti determina- 
30 tion ctosed curve as an ideal line having no widtti from such designatk)n line having wkfth. a line (cunre) whkrfi mini- 
mizes square error of distances from ttie centers of pixels constituting ttie designation line (portions indicated by mark 
X in Fia 32B) by ttie least square (mettwd) to allow ttiis line to be ttie patti determination cfosed curve. 

This patti determination dosed curve is delivered from ttie line ttiinning section 61 to ttie feature point determining 
section 22. 

35 It is to be noted ttiat ttie mettxxJ of ttiinning the designation line is not limited to ttie above^escribed mettiod. but 
ottier mettiods may be used for tfiis purpose. Namely, e.g., an approach may be enptoyed to remove pixels constituting 
the designatton line from pixels positioned outskJe to ttiereby generate a tiiin line having widtti consisting of one pixel 
to further determine a line which most satisfactorily approximate a line passing ttvough ttie centers of pixels constituting 
that ttiin line to allow ttiis line to be ttie path determination cfosed curve. 

<o , In ttie key signal generating apparatus constituted as described above, when designation line con-esponding to ttie 
boundary between ttie foreground F and ttie background B (pixels constituting ttie boundary) as ttie result of ttie fact 
that ttie operation section 1 is operated by user, ttiinning of ttiat designation line is earned out at ttie line thinning section 
61 of the feature point detecting section 60. whereby ttie patti determination closed curve is determined. The patti 
determination closed curve ttius obtained is delivered to the feature point determining section 22. At times subsequent 

45 thereto, processing similar tiD ttie key signal generating apparatus shown in FIG. 10 is carried out. Thus, a key signal a 
corresponding to ttie entirety of ttie pcture C is determined. 

Accordingly, since ttie patti determination dosed curve is detennined on ttie basis of designation by user (designa- 
tion line) in ttiis case, change C* of ttie pixel value becomes unnecessary in ader to determine ttie patti determination 
dosed curve. Namely, in ttiis case, ttiere is no necessity of delivering change C of ttie pixel value witti respect to ttie 

so feature point detecting section 60 from the change detecting section 10. 

While ttie key signal generating apparatus shown in FIG. 10 is adapted to determine difference F-B at ttie F-B 
detecting section 30 to normalize ttie change C of ttie pixel value ttiereafter to determine ttie key signal, ttie key signal 
generating apparatus may be of construction in which no F-B detecting section 30 is provided. 

As a more practical example, ttie configuration of anottier embodiment of ttie key signal generating apparatus to 

55 which ttiis invention is applied is shown in FIG. 33. It is to be noted ttiat, in ttie key signal generating apparatus shown 
in FIG. 33. ttie same reference numerals are respectively attached witti respect to portions corresponding to ttie key 
signal generating apparatus shown in FIG. 10. and ttieir explanation will be omitted as ttie occasion may demand. 
Namely, this key signal generating apparatus is adapted so ttiat ttie F-B detecting section 30 is omitted and an integral 
value cateulating section 70 is provkJed in place of tti integral value calculating section 40. In addition, ottier portions 
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of the key signal generating apparatus are constituted in the same manner as in the key signal generatina aooaratus 
shown in FIG. 10. ^ ^ \f «iuo 

The integral value calculating section 70 is operative so that, within the range where quantity (magnitude) of 
change C* of the pixel value is above a predetermined threshoW value K (hereinafter referred to as integral range as the 
occasion may demand), it integrates its change quantity, e.g.. along the direction in which change of the pixel value C 
has been detected, or the like to calculate a key signal (the skeleton portion of the key signal) 

Namely, the integral value calculating section 70 first determines points on the boundary of the gray area i e 
points Pi . P2 where a is equal to 0. 1 of pants on. e.g.. the maximum inclination (gradient) direction path U (It is to be 
noted that the path itself may be path except for the maximum inclination (gradient) direction path U as described 
above), in more practical sense, as shown In FIG. 34A. for example, the integral value calculating section 70 determines 
two points where magnitude of change C of the pixel value delivered from the change detecting section 10 becomes 
equal to the predetermined threshold value K to allow them to be respectively points p, . pg (for example when observ- 
ifig the change C of the pixel value from the background B side toward the foreground F side, the point where the 
change C of the pixel value becomes equal to the threshoW value K In the beginning is caused to be the point p. and 
the point where the change C of the pixel value becomes equal to the threshoW value K thereafter is caused to be the 
point pg.). 

Further, the integral value calculating section 70 calculates, in accordance with the following equation (19) integral 
value (definite integral value) D (definite integral informatton) of change of the pixel value C within the range (Integral 
range) from the point to the point p2, e.g., along the maximum indination (gradient) direction path U. 

0^jC'[p)dp (19) 
pi 



Then, the integral value calculating sedfon 70 normalizes the change C of the pixel value by the definite integral 
value D to thereby determine normalized value C'/D to integrate this namalized value C'/D. e.g.. in accordance with the 
tolkjwing equatk)n (20) to thereby determine a key signal a with respect to the pixel at the point (positfon) p. 




pi 



The definite integral value D corresponds to the absolute value IF-Bl of difference F-B in the case of the key signal 
generating apparatus shown in FIG. 10. Accordingly, when the change C of the pixel value is normalized by the definite 
integral value 0 to carry out the integratk>n by the equatjon(20). it is possible to easily obtain a key signal a which takes 
value vrtthin the range from 0 to 1 . It is to be noted that while, in this case, the key signal a is obtained within the range 
from the points (positfons) pi to p2 (the portion labeled "effective (valid) wWth" in FIG. 348). values of key signals within 
the range at the left sWe (positkmed) relative to the point Pi or the range at the right sWe (positioned) relative to the 
point in FIG. 348 are caused to be respectively 0 and 1 (interpolation of 0 and 1 is camed out at the key signal gen- 
erating sectfon 50 as descrS>ed above). 

Since it is unnecessary to provWe the F-B detecting sectton 30 at this key signal generating apparatus from facts 
as described above, the apparatus can be of a smaller-size configuration as compared to the key signal generating 
apparatus shown in FIG. 1 0. 

The operation of the key signal generating apparatus shown In FIG. 33 will now be desaibed with reference to the 
flowchart shown in Fia 35. 

At step S21 . the change detecting section 10 cafoulates change C* of the pixel value between pixels of picture (input 
pkrture) C similarly to the case of the step SI of FKl 1 1 to deliver it to the feature point detecting sectton 20 and the 
integral value cateulating section 70. 

At step S22. the feature point detecting sectfon 20 is operative so that when the change C* of the pixel value is deliv- 
ered thereto from the change detecting section 10. it detects the path detennination closed curve and the feature points 
similarly to the case at the step S2 of FIG. 1 1 to deliver the path determination closed cun^e and the feature points to 
the integral value calculating sectfon 70. 

At step S23. the integral value calculating section 70 is operative so that the change C of the pixel value is delivered 
thereto from the change detecting section 1 0 and the path determinatkNi dosed curve and the feature points are deliv- 
ered thereto from the feature point detecting section 20. it deterntines the integral path. e.g.. similarly to the integral 
value calculating sectfon 40 of FIG. 10 to further determine points pi. p2 where magnitude of change C of the pixel 
value becomes equal to a predetennined threshold value K. as has been explained with reference to FIG. 34A. on the 
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integral path. Then, the Integral value calculating section 70 calculates integral value D of the quantity of the change C 
of the pixel values at the points on the integral path ensting within the range from the point p, to the point p, in accord- 
ance with the equation (19). r-i r 1-^ WW 

At Step S24. the integral value calculating section 70 normalizes the change C of the pixel value by the inteoral 
value D to calculate normalized value CVD. / ^cu 

At step S25. the Integral value calculating section 70 determines, in accordance with the equation (20). the key sig- 
nal with respect to the points p constituting the integral path (path segment), i.e.. the skeleton portion of the key signal 
to deliver it to the key signal generating section 50. Further, the integral value cateulating section 70 generates Pi 
closed curve and P2 dosed curve similarly to the integral value calculating section 40 of FIG. 10 to deliver them to the 
key signal generating section 50. 

At step S26. the key signal generating section 50 carries out processing similar to that in the case of the step S6 of 
FIG. 1 1 to thereby generate a key signal a corresponding to the entirety of the picture C. Tlius, the processing at the 
key signal generating apparatus is completed. 

An example of the configuration of the integral value calculating section 70 of FIG. 33 is shown in FIG. 36. 

Path determination closed curve and feature points from the feature point detecting section 20 are delivered to an 
integral path determining section 71 . and change C of the pixel value from the change detecting section 10 is delivered 
to ttie integral path determining section 71. a definite integral value calculating section 72. and a normalization execut- 
ing section 73. In this example, the integral path determining section 71 is also supplied with a predetermined threshoW 
value K from circuit (not shown). 

The integral path determining section 71 determines integral paths passing through respective feature points, sim- 
ilarly to the integral path determining section 41 shown in FIG. 24. on the basis of the feature points and the path deter- 
mination closed ounces to deliver these integral paths to the definite integral value calculating section 72. the 
normalization executing section 73 and the integration executing section 74. Further, the integral path determining' sec- 
tion 71 detenrtnes points where the magnitude of the change C of the pixel value becomes equal to the predetermined 
threshoW value K. i.e.. points Pi . Pa of points on the integral paths passing through respective feature points to deliver 
these points Pi . P2 to the definite integral value calculating section 72. an integration executing section 74 and a dosed 
curve generating sectk)n 75. 

The definite integral value cateulating section 72 performs an operation in accordance witii the equation (19). 
Namely, ttie deJinite integral value calculating section 72 carries out d^inite integral of change C* of the pixel value 
within the range from the point pi up to the point P2 similarly delivered from the integral path determining section 71 
atong the inte^ paths passing through respective feature points delivered from the integral path determining section 
71 to deliver, to the normalization executing section 73, integral values (definite integral values) D every respective inte- 
gral paths obtained as the result thereof. 

The normalization executing section 73 normalizes change C of the pixel value on each integral path delivered 
from the integral path determining section 71 by the definite integral value D with respect to each integral path delivered 
from tiie d^lnlte integral value calculating section 72 to deliver the nomialized value C/D obtained as the result thereof 
to ttie integration executing section 74. 

The Integration executing section 74 performs an operation in accordance with the equation (20). Namely, the inte- 
gration executing section 74 carries out integration of the normalized value C'/D delivered from the normalization exe- 
cuting section 73 atong each integral path delivered from ttie integral patfi detemiining section 72 to output the skeleton 
portion of the key signal as the integral value on each Integral patti (paWi segment) obtained as the result thereof. It is 
to be noted that while the integration along each integral path at the integration executing section 74 is carried out 
according to the equation (20) in such a manner that the point p, on each integral path delivered from the integral path 
determining sectton 71 is caused to be the initial (start) point, such integration may be also carried out in such a manner 
that the point pg is caused to be the initial point. 

The dosed curve generating sectton 75 is operative so that when points Pi , pg on ail integral paths are delivered 
thereto from the integral path determining section 71 . it connects the points p, to each ottier. and connects the points 
Pa to each other to thereby respectively generate Pi dosed cun^e and P2 dosed cun/a 

The operation of the integral value calculating sectton 70 will now be described vwth reference to the flowchart of 
FIG. 37. 

At step S21-1 , the integral path determining section 71 carries out processing similar to that in the case of the step 
S4-1 of FIG. 26 to determine integral paths passing through the renwked feature points to deliver these integral paths 
to the definite integral value cateulating section 72. the normalization executing section 73 and the integratton executing 
section 74. 

At step S21-2, the integral path determining section 71 detects points pi, pa where the magnitude of change C of 
pixel value delivered from the change detecting section 10 becomes equal to the predetennined threshold value K of 
points on the integral paths passing through the remarked feature points to deliver these points Pi, pg to the definite 
integral value calculating section 72. the Integration executing section 74 and the closed cun/e generating section 75. 

At step S21-3. the definite integral value calculating section 72 is operativ so that when the integral path with 
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respect to the remarked feature point and points Pi. P2 on the integral path are delivered thereto from the integral path 
determining section 71 . it carries out integration of change C of the pixel value within the range from the point Pi to the 
point P2 aJong the integral path in accordance with the equation (19) to deliver definite integral value D obtained as the 
result thereof to the normalization executing section 73. 

At step S21-4. the normalization executing section 73 normalizes the change C* of the pixel value of pixel on the 
integral path with respect to the remarked feature point by the definite integral value 0 delivered from the definite inte- 
gral value calculating section 72 to deliver normalized value C7D obtained as the result thereof to the integration exe- 
cuting section 74. " 

At step S21 -5. the integration executing section 74 integrates, in accordance with the equation (20). the normalized 
value C7D delivered from the normalization executing section 73. with. e.g.. the point pi delivered from the integral path 
determining section 71 being as the initial (start) point, along the integral path similarly delivered from the integral path 
determinating section 71 to calculate the key signal on the integral path. i.e.. the skeleton portion of the key signal 

At step S21 -6, the integral path determining section 71 judges whether or not all of feature points from the feature 
point detecting section 20 have been caused to be the remarked feature point In the case where the integral path deter- 
mining section 71 has judged at this step S21-6 that all of feature points from the feature point detecting section 20 have 
not yet been caused to be the remarked feature point the processing returns to the step S2M . Thus, the integral path 
determining section 71 allows the feature point which is not yet caused to be the remarked feature point to be newly the 
remarked feature point At times subsequent thereto, processing as described above is repeated. 

On the other hand, in the case where the integral path determining section 71 has judged at the step S21 -6 that all 
of feature points have been caused to be the remarked feature point i.a. integration-of the equation (20) along ail tiie 
integral paths passing tiirough respective feature points delivered from the feature point detecting section 20. whereby 
key signals on those integral paths (the skeleton portions of ttie key signals) have been determined, the processino oro- 
ceedstostepS2l-7. ^ 

At the step S21 -7. the ck>sed curve generating section 75 generates PI closed curve and P2 dosed cun/e Thus 
the processing at the integral value calculating section 70 is conpleted. 

Namely, the closed curve generating section 75 connects the points Pi to each other and connects the points Pa to 
each other, those points existing on all integral paths delivered from the integral path determining section 71 to ttiereby 
respectively generate Pi dosed curve and P2 dosed curve. 

It is to be noted ttiat ttie integration at the integration executing section 74 is can-ied out only with respect to the patfi 
segment of the integral patti similarly to ttie case at ttie integration executing section 43 shown in FKjL 24. 

FIG. 38 shows ttie corrflguration of an embodiment of a picture synttiesis apparatus to whkrfi ttiis invention is 
applied. 

Picture providing units conprise. ag. . a vWeo tape recorder, hard dsc devfce or disc player, etc.. and serve to 
reproduce a predetermined picture to output it In this example, picture T1 (first picture) in whk^ the foreground and ttie 
background that ttie picture providing unit 81 reproduces are caused to be respectively F and B is delivered to a key 
signal generating unit 84 and a mixer 85. Thus, ttiis picture T1 is caused to be ttie object pidure for generating ttie key 
signal. Moreover, picture T2 (second picture) that ttie picture providing unit 82 reproduces is delivered to the mixer 85. 
at which ttie foregrowid F of ttie picture Tl is synthesized with respect to ttie pknure T2, Furttier. picture T3 obtained 
by photographing or picking up image of only ttie background B of the pk:ture Tl that ttte picture provicfing unit 81 repro- 
duces is outputted from ttie picture provkJing unit 83. Namely, when, e.g.. ttie picture Tl is assumed to be ttie picture 
iriiage where a person is walking within a certain scene, ttie pcture T3 in which image of ttiat scene (background B) is 
picked up in ttie state where ttiat person (foreground F) is exduded is outputted from ttie picture provkfing unit 83. The 
picture T3 ttiat ttie pteture provkSng unit 83 reproduces is delivered to ttie mixer 85. 

The key signal generating unit 84 is constituted in a manner similar to ttie key signal generating unit of FIQ. 10. 30 
or 33. and generates a key signal a as described above to deliver it to ttie mixer 85. The mixer 85 synttiesizes ttie (Die- 
ture Tl and ttie picture T2 Inputted thereto by using ttie key signal a from ttie key signal generating unit 84. Namely, ttie 
mixer 85 extracts ttie foreground F from ttie picture Tl to synthesize (incorporate) ttiat foreground F into (witti respect 
to) the picture T2. 

The operation of ttie picture synttiesis apparatijs will now be described. 

At the picture provWIng units 81 to 83. pictures Tl to T3 are respectively reproduced. These ptetures Tl to T3 are 
delivered to ttie mixer 85. Furttier. ttie pkrture Tl ttiat ttie picture providing unit 81 reproduces is delivered also to ttie 
key signal generating unit 84. 

The key signal generating unit 84 generates a key signal a in a manner as described above to deliver it to ttie mixer 
85. The mixer 85 extracts ttie foreground F from ttie picture Tl inputted thereto by using ttie key signal a to further syn- 
thesize ttiat foreground F into (with respect to) ttie picture T2 to generate picture in which ttie foreground F is synttie- 
sized into (witti respect to) ttie pk:ture T2 (synttietic pk:ture) T4. 

Namely, assuining now that pixel values of pixels at respective positions p constituting respective pictijres T1 . T2 
are respectively designated at P(p), Q{p), pixel values of pixels at respective positions p constituting respective fore- 
ground and background of ttie picture Tl ar respectively designated at F(p), B(p), and pixel value of pixel at each posi- 
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(22) how"^^"^ ^® synthetic picture T4 is designated at R(p). the following equations (relational expressions) (21). 

P(p) = aF(p) + (1.a)B(p) (2i) 

R(p) = aF(p) + (1.a)Q(p) (22) 

When aF(p) of the right side of the equation (22) is eliminated from the equation (21), the equation (22) is rewritten 
into the following equation (23) 

R(P) = P(P) - (1 'Ci)B(p) + (1 .a)Q(p) (23) 

At the mixer 85. synthetic picture T4 is generated in accordance with the equation (23). Namely, pixel values P(p). 
Q(P). B(p) are respectively delivered from the picture providing units 81 to 83 to the mixer 85. and the key signal a(p) is 
delivered from the key signal generating unit 84 also to the mixer 85. At the mixer 85. operation of the equation (23) Is 
performed by using the respective pixel values and the key signal. Thus, picture T4 consisting of pixel value R(p). i.e.. 
in whteh the foreground F is synthesized with respect to the picture T2 is generated. 

This picture T4 Is delivered to. e,g.. recording unit (e.g., video tape recorder, hard disc device, or writaiDle disc drive 
eta), at which It Is recorded. 

Since precise key signals, i.e.. key signals in which the influence of motion tiiur Is taken Into consideration and alias 
IS permitted to be reduced are outputted from the key signal generating unit 84 as described above, in the case where 
such key signals are used to carry out synthesis of picture, precise synthetic pkAire (picture free from disagreement of 
feeling when viewed) can be obtained. 

It is to be noted that the picture synthesis apparatus shown in FIG. 38 may be constituted as desc^toed below. 
Namely, since background B is provkled by the pk^ire providing unit 83. in the case where the key signal generating 
unit 84 is constituted by such a key signal generating apparatus as shown in FIG. 10 or 30 including the F-B detecting 
section 30. background B is delivered to the key signal generating unit 84 as indicated by dotted lines in FIG. 38. thus 
making It possible to determine difference F-B from the background B In the key signal generating unit 84. 

Meanwhile, since pteture T2 consisting of pixel value Q(p) is obtained from the picture provkiing unit 82 and the key 
signal a Is obtained from the key signal generating unit 84. If the pixel value F(p) of the foreground F can be recognized 
(the pixel value F(p) can be estimated), it is possible to generate, also by the equation (22). picture T4 in whfch the fore- 
ground F is synthesized with respect to ttie picture T2. 

In view of the above, the configuration of an embodiment of the picture synthesis apparatus adapted for generating 
picture T4 In accordance with the equation (22) is shown in FIG. 39. ft is to be noted that the same reference numerals 
are respectively attached with respect to portions corresponding to the picture synthesis apparatus shown in FIG. 38 in 
the picture syrthesis apparatus shown In FIG. 39, and their explanation will be omitted as the occasion may demand. 
Namely, in this picture synthesis apparatus, an information generating unit 90 and a mixer 96 are provWed in place of 
the pfoture provkiing unit 83 and the mixer 85. In addition, other portions of the pkAire synthesis apparatus are consti- 
tuted similarly to the picture synthesis apparatus of FIQ. 38. 

The infomiation generating unit 90 is supplied with pkAjre T1 that the picture provkiing unit 81 outputs. The Infor- 
mation generating unit 90 estimates (generates), from the pictijre Ti . pixel value F(p) of the foreground F thereof to 
deliver it to the mixer 96. The mixer 96 perfbrnre an operation In accordance with the equation (22) by using pkrtures 
TI, T2 (pixel values P(p), Q(p)) that the pkrture Infomiation providing units 81 , 82 respectively output, key signal a(p) 
that the key signal generating unit 84 outputs, and foreground F (pixel value F(p)) that the information generating unit 
90 outputs to ttiereby generate synthetic picture T4 in which the foreground F is synthesized with respect to the pfoture 

In the pictu'e synthesis apparatus constituted as described above, pictures T1 , T2 are respectively reproduced at 
the picture providing units 81. 82, These pfotures T1 , T2 thus reproduced are delivered to the mixer 96. Further, the pic- 
ture TI that the picture provkiing unit 81 reproduces Is delivered also to the information generating unit 90 and the key 
signal generating unit 84. The key signal generating unit 84 generates the key signal a in a manner as described above 
to deliver it to the ntixer 96. 

On tiie other hand, the information generating unit 90 determines, from the picture TI , pixel value F(p) of ttie fore- 
ground F thereof. 

A more practical example of the configuration of the infomnation generating unit 90 is shown in FKS. 40. This infor- 
matton generating unit 90 is composed of a change detecting section 91 , a boundary detecting section 92. a very small 
area detecting section 93, a color distribution forming section 94. and a center of gravity detecting section 95. The 
change detecting section 91 is constituted similariy to the change detecting sections 10 respectively shown in FIGS. 10. 
30 and 33. and the boundary detecting section 92 is constituted sinrdlariy to the boundary detecting section 21 of FiQ. 
1 6. Moreover, th very small area detecting section 93, the cola distributfon fomfiing section 94 and the center of gravity 
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detecting section 95 are constituted respectively similarly to the very small area detecting section 31 . the color distribu- 
tion forming section 32F and the center of gravity detecting section 33F. It is to be noted that the very small area detect- 
ing section 93 extracts only pixels constituting the foregrounds F with respect to respective very small areas to deliver 
pixel values of those pixels, i.e.. only F boundary pixel values to the color distribution forming section 94. 

Accordingly, at this information generating unit 90. change P of the pixel value P of picture Tl is determined at the 
change detecting section 91. and path determination curve and feature points are determined on the basis of the 
change P' at the boundary detecting section 92. Further, at the very smail area detecting section 93. very small area 
including respective feature points is detected, and F boundary pixel values are extracted with respect to respective 
very small areas. Further, color distribution on the color space of F boundary pixel values in respective very small areas 
»s determined at the color distribution forming section 94. In addition, center of gravity of the color cfistrft)ution is deter- 
mined at the center of gravity detecting section 95. and the center of gravity thus obtained is outputted as pixel value 
F(p) of tiie foreground F of pixels existing at the periphery of respective feature points in the gray area 

In this example, at the information generating unit 90. with respect to the portion except lor the gray area of the fore- 
ground F (completely the portion of the foreground F), pixel value F(p) of tiie foreground F thereof is outputted as it is. 

Turning back to FIG. 39. at the mixer 96. pixel value R(p) of picture T4 in which the foreground F is synthesized with 
respect to the picture T2 is determined in accordance with the equation (22) by using pixel values P(p), Q(p) that the 
picture information providing units 81 . 82 respectively output, key signal a(p) tiiat the key signal generating unit 84 out- 
puts, and pixel value F(p) that the information generating unit 90 outputs. 

Accordingly, also in this case, it Is possible to obtain precise symhetic picture. 

In this instance, at the information generating unit 90. background B may be generated in place of generating tore- 
ground F. In this case, at the mixer 96. pixel value R(p) of synthetk: picture T4 may be determined in accordance with 
the equatfon (23) by using the background B generated by ttie information generating unit 90. 

It is to be noted that the synthesis method for picture at the mixers 85. 96 are not limited to the above-described 
method, but other syrrthesis methods may be used. 

Further, while integration is carried out at the integral value cafoulating sections 40. 70 in this errtxxliment. there 
may be emptoyed. In place of tine integration, an operation equKmlent to the integratfon. i.e. an operatfon to set sanple 
points at the path for canying out integration to add sample values at respective sanple points. 

In addition, while key signal portion except for the skeleton portion of key signals in the gray area is determined by 
interpolation In this embodiment, key signal portion except for the skeleton portion of the key signal may be determined, 
e.g.. by carrying out integration based on the equatfon (18) <x the equation (20) in place of the interpolation. 

It shouW be noted that various modifteations and/or application examples are conceivable (applicable) witiiin the 
range which does not depart from the gist of this Invention. Acooftiingly. the gist of this Invention is not limited to the 
errtediments. 



Industrial Applicability 



In accordance with the key signal generating apparatus and the key signal generating method according to this 
invention, change of pixel value between pixels constituting pfoture is detected to generate a key signal on the basis of 
change of that pixel value. Accorcfi ngly. precise key signal can be obtained. 

In accordance witii the picture synthesis apparatus and tiie picture synthesis method according to this invention, 
change of pixel value between pixels constituting a first picture is detected to generate a key signal on the basis of the 
change of that pixel value. Then, that key signal is used to synthesize the foreground picture of tiie f list pfoture and the 
background picture of a second picture. Accordingly, precise synthetic picture can be obtained. 

Claims 

1 . A key signal generating apparatus adapted for generating a key signal from pfoture. 

ttie apparatus comprising: 

change detecting means for detecting change of pixel value between pixels constituting the picture: and 
key signal generating means for generating the key signal on ttie basis of ttie change of the pixel value output- 
ted from ttie diange detecting means. 

2. A key signal generating apparatus as set fortti in claim 1 , 

wherein the key signal generating means is adapted so ttiat in tiie case where motion blur takes place in ttie 
picture, it generates ttie key signal in which ttiat motion blur is reflected. 

3. A key signal generating apparatus as set forth in claim 1. 

wherein the key signal generating means is adapted so that in the case where alias takes place in ttie pic- 
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ture. it generates the key signal for allowing the influence of that alias to be reduced. 

4. A key signal generating apparatus as set forth in daim 1 , 

wherein the picture consists of foreground picture and background picture. 

wherein the apparatus further comprises difference detecting means for detecting difference between the 
foreground picture and the background picture to output difference information indicating that difference 

wherein the change detecting means outputs change information indicating change of the pixel value and 
wherein the key signal generating means generates the key signal on the basis of the difference information 
and the change information. 

5. A key signal generating apparatus as set forth in daim 1 . 

wherein the key signal generating means integrates change information indk^ting change of the pixel value 
to generate the key signal. 

6. A key signal generating apparatus as set forth in daim 4. 

wherein the key signal generating means comprises: 

normalizing means for normalizing the change information by the difference infbmiatfon to output normalized 
information obtained as the result of the normalizatk)n» and 

integration executing means fbr integrating the normalized information along a predetermined integral path. ' 

7. A key signal generating apparatus as set forth in daim 6, 

wherein the predetermined integral path is a path along a direction in which change of the pixel value has 
been detected by the change detecting means. 

8. A key signal generating apparatus as set forth in daim 1 , 

wherein the change detecting means outputs change information indicating change of the pixel value, and 
wherein the key signal generating means integrates the change informatiwi within an integral range i^vhich 

IS the range where the change information takes a predetermined threshold value or more to ttiereby generate the 

key signal. 

9. A key signal generating apparatus as set forth in daim 8. 

wherein the key signal generating means integrates the change information atong a predetermined integral 

path. 

10. A key signal generating apparatus as set forth in daim 9. 

wherein the predetermined integral path is a path atong a direction in which change of the pixel value has 
been detected by the change detecting means. 

11. A key signal generating apparatus as set forth in daim 8, 

wherein the key signal generating means comprises: 

range detecting means for detecting the integral range; 

definite integral information generating means for integrating the change informatfon along a predetermined 
integral path within the integral range to generate definite integral information of the change information within 
the integral range; 

normalizing means for normalizing the change information by the definite integral information to output normal- 
ized informiition obtained as the result of the normalization: and 

integration executing means for integrating the normalized information to thereby generate the key signal. 

12. A key signal generating apparatus as set forth in daim 1 1. 

wherein the predetermined integral path is a path along a direction in which change of the pixel value has 
been detected by the change detecting means. 

13. A picture syrthesis apparatus adapted for receiving a first picture consisting of at least foreground picture and 
background picture, and a second picture consisting of at least background picture to syntiiesize the foreground 
picture of the first picture and the background picture of the second picture. 

the apparatus comprising: 
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change detecting means for detecting change of pixel value between pixels constituting the first picture: 
key signal generating means for generating a key signal on the basis of the change of the pixel value outputted 
from the change detecting means: and 

synthesis means for synthesizing the foreground picture of the first picture and the background picture of the 
second picture by using the key signal generated by the key signal generating means. 

14. A picture synthesis apparatus as set forth in claim 13. 

which further comprises difference detecting means for detecting difference between the fbreground picture 
and the background picture of the first picture to output difference information indicating the difference. 

wherein the change detecting means outputs change. information indicating change of the pixel value and 
wherein the key signal generating means generates the key signal on the basis of the difference information 
and the change information. 

15. A picture synthesis apparatus as set forth in claim 13, 

wherein the change detecting means outputs change infonnation indicating change of the pixel value, and 
wherein the key signal generating means integrates the change informalfon within an integral range which 
is the range where the change infomiatfon takes a predetermined threshold value or mora 

16. A key signal generating method of generating a key signal from pkrture, 

the method comprising the steps of: 

detecting change of pixel value between pixels constituting the pidure: and 
generating the key signal on the basis of the change of the pixel value. 

17. A key signal generating mettwd as set forth in daim 16, 

wherein in the case where motton Uur takes place in the pkrture, the key signal In which that motion blur is 
reflected is generated. 

18. A key signal generating method as set forth in daim 16, 

wherein in the case where alias takes place In the pfoture. the key signal fbr alkywing the influence of the 
alias to be reduced is generated. 

19. A key signal generating method as set forth in daim 16. 

wherein the picture consists of foreground picture and background picture, 
the method comprising the steps of: 

detecting difference between the foreground picture and the background picture to output different information 
indicating the difference: and 

generating the key signal on the basis of change information indicating change of the pixel value and the dif- 
ference infomiatfon. 

20. A key signal generating method as set forth in daim 1 6. 

wherein change infbrmatfon indcating change of the pixel value is integrated to thereby generate the key 

signal. 

21. A key signal generating method as set forth in daim 19, 

the method comprising tiie steps of: 

normalizing the change information by the difference infomiatfon to output normalized information obtained as 
the result of the normalization: and 

integrating the normalized infbrmatfon along a predetermined integral path to thereby generate the key signal. 

22. A key signal generating method as set forth in daim 21 , 

wherein the predetermined integral path is a path along a direction in which change of ttie pixel value has 
been detected. 

23. A key signal generating mettwd as set forth in daim 16, 

wherein, within an integral range which is the range where change infamation indkating change of the pixel 
value takes a predetemiined threshokJ value or mor . the change information is integrated to thereby generate the 
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key signal. 

24. A key signal generating method as set forth in claim 23. 

wherein the change information is integrated along a predetermined integral path to thereby generate the 
key signal. 

25. A key signal generating method as set forth in claim 24. 

wherein the predetermined integral path is a path along a direction in which change of the pixel value has 
been detected. 

26. A key signal generating method as set forth in claim 23. 

the method comprising the steps of: 

detecting the integral range; 

integrating the change information along a predetermined integral path within the integral range to generate 
definite integral information of the change information within the integral range; 

normalizing the change information by the definite integral infomiatlon to output normalized information 

obtained as the result of the normalization; and 

integrating the nomialized information to thereby generate the key signal. 

27. A key signal generating method as set forth in claim 26. 

wherein the predetermined integral path is a path along a direction in which change of the pixel value has 
been detected. 

28. A picture synthesis method of receiving a first picture consisting of at least foreground picture and backyound pic- 
ture and a second pk;ture consisting of at least background picture to synthesize the foreground picture of the first 
picture and the background picture of the second picture, 

the method comprising the steps of: 

detecting change of pixel value between pixels constituting the first picture; 
generating a signal on the basis of the change of the pixel value; and 

synthesizing the foreground picture of the first pwture and the bad<ground picture of the second picture by 
using the key signal. 

29. A picture synthesis method as set forth in claim 28. 

the method comprising steps of: 

detecting difference between the foreground picture and the background picture of the first picture; and 
generating the key signal on the basis of difference information indicating the difference and change informa- 
tion indicating change of the pixel value. 

30. A picture synthesis method as s^ forth in claim 28. 

wherein, within an integral range wWch is the range where change infomiation indicating change of the pixel 
value takes a predetermined threshoU voltage or more, the change information is integrated to thereby generate 
the key signal. 
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Description 
Technical Field 

[0001] This invention relates to a key signal generating apparatus and a picture synthesis apparatus, and a key 
signal generating method and a picture synthesis method. More particularly, this invention relates to a key signal gen- 
erating apparatus and a picture synthesis apparatus, and a key signal generating method and a picture synthesis 
method which are adapted for detecting change of pixel value between pixels constituting pictures to generate a key 
signal on the basis of the change of the pixel value, thereby making it possible to obtain an accurate key signal. 

Background Art 

[0002] For example, in movies and other image productions, in the case where various simulations of pictures (im- 
ages) are carried out, or any special effect is given to pictures, a remarked portion (object) is extracted from picture, 
or the extracted portion is synthesized with respect other pictures. Such a partial extraction from picture and/or synthesis 
(of the extracted portion) is carried out by using key signal. 

[0003] As a technology relating to such a key signal, an example thereof is proposed in the specification and the 
drawings of the U.S. Patent No. 4.488,169 (Registered Date: December 11 . 1984) by the applk:ant of this invention. 
[0004] As the key signal, there are a key signal called hard key and a key signal called soft key. Now. in the case 
where, e.g., the remarked portion within picture used for generating the key signal is caused to be the foreground 
(foreground picture), and the portion except for the above is caused to be the background (background picture), a 
binary key signal (mask picture) in which the range of the foreground in the pteture is caused (assumed) to be 1 and 
the range of the background is caused (assumed) to be 0 as shown in FIG. 1 A is called hard key. On the contrary, a 
key signal whksh can take not only two values of 0 and 1 , but also continuous real numbers within the range from 0 to 
1 as shown, in FIG. IB in consideration of alias and/or motion blur which will be described later taking place in picture 
is called soft key. Here, the area where the key signal takes values within the range from 0 to 1 1s called gray area as 
the occasion may demand. 

[0005] Namely, it can be said that the hard key is a signal having a sharp boundary (border) line (signal in which 
inclination (gradient) of the boundary line where the key signal changes from 0 (orl ) to 1 (or 0) is sharp), and the soft 
key is a signal having a smooth (gentle) boundaiy (border) line (signal In whteh inclination (gradient) of the boundary 
line where the key signal changes from 0 (or 1) to 1 (or 0) is smooth (gentle)). 

[0006] It Is now assumed that a (o value) shown in FIGS. 1 A and 1 B indicates continuous key signal values and also 
indk;ates, in this case, values of the key signal of pixel units at the same time. In practtee, e.g.. continuous key signals 
are caused to undergo filtering by the range of pixel, etc. to use, as the key signal of the pixel unit, the result by that 
filtering at the center of corresponding pixel. In this case, in the picture, pixel value of the pixels at the boundary portion 
between the foreground and the background is a value in which respective signal components of the foreground and 
the background are interposed. From facts as above, it can be said that a indicates contribution ratio (factor) indicating 
to what degree the foreground contributes with respect to respective pixels. 

[0007] Meanwhile, it is rare that in the case where both the foreground and the background are stationary, the bound- 
ary between the foreground and the background is positioned at the boundary line between pixels constituting a cor- 
responding picture. Ordinarily, as shown in F|G. 2A, the boundary between the foreground and the background exists 
within the pixel having finite dimensions. Howeven since the pixel is the minimum unit constituting the picture, even if 
the boundary between the foreground and the background as described above exists within the pixel, that pixel is 
caused to have color or brightness (pixel value) of any one of the foreground and the background as shown in FIG. 
2B. For this reason, the boundary line between the foreground and the background is drawn along the boundary line 
between pixels as shown in FIG. 2C. When the entirety of such a picture is viewed, unnatural flteker, i.e., alias takes 
place at the portion of the boundary between the foreground and the background (boundary line between pixels). 
[0008] Accordingly, even if a key signal smoothly indicating the boundary line between the foreground and the back- 
ground can be obtained, alias would take place in the synthetic pteture obtained as the result of the fact that the key 
signal mentioned above is used to cany out picture synthesis. 

[0009] In view of the above, as a method of preventing alias taking place in the synthetic picture, there are, e.g.. 
methods disclosed in the Japanese Patent Applteatlon Laid Open No. 232787/1985 and the Japanese Patent Appli- 
cation Laid Open No. 340671/1 992. In the case of the method disclosed in the Japanese Patent Application Laid Open 
No. 232787/1985. the hard key is used to carry out picture synthesis, and to apply low-pass filter (LPF) only to the 
boundary portion between the foreground and the background to allow the boundary portion thereof to be blun-ed to 
allow it to be smooth to look at. In this method, since the hard key is used, but LPF is applied to the boundary portion 
between the foreground and the background, It can be said that picture synthesis is earned out by using key in which 
the key signal is caused to have gradient (inclination), i.e.. substantially soft key. 
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[OOIOJ Moreover, in the method disclosed in the Japanese Patent Application Laid Open No. 340671/1992 a block 
in which a remarked pixel on the boundary between the foreground and the background is caused to be the center is 
set to calculate a key signal (soft key) from the pixel pattern constituting the foreground and the background within that 
block to carry out, by using such a key signal, extraction of the foreground, and synthesis of the foreground and the 
background to thereby reduce the alias. 

[0011] However, since the soft key used in the above-mentioned method is such that value of a at the inclined 
(gradient) portion (the portion of 0 < a < 1 in FIG. (B)). i.e., in the gray area is uniform, there was the problem that it is 
difficult to cope with motion blur taking place, e.g.. in the case where the foreground is moving. 
[0012] The motion blur will now be briefly described. The moving picture is realized as the result of the fact that 
picture (still picture) is continuously displayed, e.g.. In frame units, etc. For example, assuming now that a circular 
object moving from the left direction to the right direction within picture exists, in the case where the frame period is 
assumed to be a short time such that it can be considered to be infinitesimal (infinitely small) with respect to the 
movement velocity of that object, that object is displayed as a circular object in respective frames as shown in FIG. 
3A. However, in the case where the frame period Is not a short time such that it can be considered to be infinitesimal 
wrth respect to the movement velocity of the circular object, picture of frame at a certain time tg becomes a picture in 
which movement of the object from time t^ of frame preceding by one to the time tj is reflected as shown In FIG. 38 
Namely, the object within the pk:ture of the frame at the time tj is not circular picture image, but picture image extended 
(elongated) in a thin manner in the direction of that motion such that the contour is blur. Such a phenomenon is called 
motion blur. 

[0013] It Is to be noted that while motion blur takes place not only In the case where any one of the foreground and 
the background is moving, but also in the case where both are independently moving, in the case where only the 
background is moving and in the case where both are independently moving, it is possible to consider those cases 
similariy to the case where only the foreground is moving if the background is taken as reference. In addition. In the 
case where motion blur takes place, clear boundary between the foreground and the background does not exist. 
[00141 As a method of generating a key signal in which motion blur is taken into consideration as described above 
there are methods, e.g.. the Japanese Patent Application Uid Open No. 153493/1993 and the Japanese Patent Ap- 
plication Laid Open No. 236347/1993. In the method disclosed In the Japanese Patent Applrcatlon Laid Open No. 
1 53493/1 993. cun-ent frame and frame preceding (eartier) by one are compared with each other to thereby determine 
sum total with respect to the entirety of frame of change quantities (magnitudes) of pixel value of pixels constituting 
the foreground of the current frame (or the frame preceding by one) and pixel value corresponding to pixels of the 
frame preceding by one (or the current frame) to determine the gray area of the soft key from the sum total value. 
However, in this method, since the soft key Is determined in dependency upon the above-described change quantity 
of pixel value (difference between pixel values of the same pixel), the direction of movement of the foreground is not 
taken Into consideration, so whether or not correct soft key corresponding to that movement can be obtained was not 
35 . clear. 

[0015] On the other hand, in the method disclosed in the Japanese Patent Application Laid Open No. 236347/1 993, 
an approach Is employed to first generate hard key to shave, at respective points on the boundary between the fore- 
ground and the background, the side surface of that hard key (side surfaces of column in whteh the portion of d = 1 Is 
caused to be the bottom surface shown in FIG. 1 A) by the elliptical cone to thereby generate soft key In this case, the 

^ long axis and the short axis of the ellipse constituting the bottom surface of the elliptical cone is determined by motion 
vector of the remarked portion and the magnitude of the edge intensity. However, in this method, the direction of the 
edge intensity is not correctly taken into consideration. For this reason, whether or not precise or correct soft key 
corresponding to movement of the foreground an be obtained was not clear similariy to the above. 
[0016] Further, there is known from JP.A-4070293 a method for cateulating the key signal in accordance with the 

« relational color change speed, wherein the back color and a subject are discriminated from each other. 

Disclosure of the Invention 

[001 7J It is an object of the present Invention to generate a key signal which is precisely coping with motion blur, and 
so permits the. Influence of alias to be reduced. 

[0018] This object Is achieved by a key signal generating apparatus, a picture synthesis apparatus, a key signal 
generating method and a picture synthesis method according to the enclosed independent claims. Advantageous 
features of the present invention are defined in the con-esponding subclaims. 
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Brief Description of the Drawings 
[0019] 

FIG. 1 A Is a view for explaining hard key, and FIG. 1 B is a view for explaining soft key. 
FIGS. 2A, 2B and 2C are views for explaining alias. 
FIGS, 3A and 38 are views for explaining motion blur. 

FIG. 4 is a view showing the state where local texture of picture is considered to be uniform. 
FIG. 5 is a view for explaining a method of calculating picture in which motion blur takes place. 
FIG. 6 is a view showing filtering (filter processing) for providing the effect of motion blur, 
FIG. 7 Is a view showing foreground F and key signal a in which the effect of motion blur is taken into consideration 
FIG. 8 IS a view showing that local texture of the foregrourid F to which the effect of motion blur has been applied 
. is considered to be unifomri. 
FIG. 9 is a view showing that alias takes place in the case where pixel has finite size (dimensions). 
FIG. 10 is a block diagram showing the configuration of a first embodiment of a key signal generating apparatus 
to whk:h this invention is applied. 

FIG. 11 is a flowchart for explaining the operation of the key signal generating apparatus of FIG. 10. 
FIG. 12 is a block diagram showing an example of the configuration of change detecting unit 10 of FIG. 10. 
FIG. 13 is a flowchart for explaining the operation of the change detecting section 10 of FIG. 12. 
^ FIG. 14 is a view showing sobel operator. 

FIG, 15 Is a view for explaining processing of step 81 -2 of FIG. 13. 

FIG. 1 6 is a block diagram showing an example of the configuration of feature point detecting section 20 of FIG. 10. 
FIG. 1 7 is a view for explaining processing of the feature point detecting section 20 of FIG. 16. 
FIG. 18 is a flowchart for explaining the operation of the feature point detecting secUon of FIG. 16. 
FIG. 19 is a block diagram showing an example of F-B detecting section 30 of FIG. 10. 
FIG. 20 is a view for explaining processing of very small area detecting section 31 of FIG. 19. 
FIG. 21 Is a view for explaining processing of difference calculating section 34 of FIG. 19, 
FIG. 22 is a flowchart for explaining the operation of the F-B detecting section 30 of FIG\ 19. 
FIG. 23 is a view for explaining processing of step SS of FIG. 11 . 

FIG. 24 is a block diagram showing an example of the configuration of integral value calculating section 40 of FIG 

10, 

FIG. 25 is a view for explaining processing of integral value calculating section 40 of FIG. 24. 
FIG, 26 is a flowchart for explaining the operation of the integral value calculating section 40 of FIG. 24. 
FIG. 27 is a block diagram showing an example of the configuration of key signal generating section 50 of FIG. 1 0. 
FIG. 28 is a view for explaining processing of the key signal generating section 50 of FIG. 27. 
FIG. 29 is a flowchart for explaining the operation of the key signal generating section 50 of FIG. 27. 
FIG. 30 is a block diagram showing the configuration of a second embodiment of the key signal generating appa- 
ratus to which this invention is applied. 

FIG. 31 is a block diagram showing an example of the configuration of feature point detecting section 60 of FIG. 30. 
FIG. 32 Is a view for explaining processing of line thinning section 61 of FIG. 31 . 

FIG. 33 is a block diagram showing a third embodiment of the key signal generating apparatus to which this in- 
vention is applied. 

FIG. 34 is a view for explaining processing of integral value calculating section 70 of FIG. 33. 
FIG. 35 is a flowchart for explaining the operation of the key signal generating apparatus of FIG. 33. 
FIG. 36 is a block diagram showing an example of the configuration of the integral value calculating section 70 of 
FIG. 33.. 

FIG. 37 is a ftowchart for explaining the operation of integral value calculating section 70 of FIG. 36. 
FIG. 38 is a block diagram showing the configuration of a first embodiment of a picture synthesis apparatus to 
which this invention is applied. 

FIG. 39 is a block diagram showing the configuration of a second embodiment of the picture synthesis apparatus 
to whk:h this invention is applied. 

FIG. 40 is a block diagram showing an example of the configuration of infomiation generating unit 90 of FIG. 39. 
Best Mode for Carrying Out the Invention 

[0020] Prior to description of preferred embodiments of this invention, in order to clarify the correspondence rela- 
tionship between respective means of the invention described in the claims and the embodiments described below, 
the features of this invention will be described in the following manner in the state where the description of corresponding 
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embodiments (which should be considered to be as examples) are added within parentheses after respective means. 
[0021] Namely, a key signal generating apparatus according to this invention is directed to a key signal generating 
apparatus adapted for generating a key signal from picture, the apparatus comprising: change detecting means (e.g., 
change delecting section 10, shown in FIG. 10, 30 or 33. etc.) for detecting change of pixel value between pixels 

5 constituting picture; and key signal generating means (e.g., integral value calculating section 40 shown in FIG. 10 or 
30, or integral value calculating section 70 and key signal generating section 50 shown in FIG. 33, etc.). 
[0022] The key signal generating apparatus according to this invention further comprises difference detecting means 
(e.g., F-B detecting section 30 shown in FIG. 10 or 30, etc.) for detecting difference between foreground picture and 
background picture constituting the picture to output difference information indicating the difference, wherein the change 

10 detecting means outputs change Information indicating change of pixel value, and the key signal generating means 
generates the key signal on the basis of the difference information and the change infonmation. 
[0023] In the key signal generating apparatus according to this invention, the key signal generating means comprises 
normalizing means (e.g., nomializatlon executing section 42 shown in FIG. 24. etc.) for normalizing the change infor- 
mation by the difference information to output normalized information obtained as the result of the normalization, and 

15 integration executing means (e.g., integration executing section 43 shown in FIG. 24, etc.) for integrating the normalized 
infonmation along a predetemiined integral path). 

[0024] In the key signal generating apparatus according to this invention, the key signal generating meains comprises 
range detecting means (e.g., integral path determining section 71 shown in FIG. 36, etc.) for detecting integral range, 
definite integral infonnation generating means (e.g., definite integral value calculating section 72 shown in FIG. 36. 

so etc.) for integrating the change Infonnation along a predetemilned Integral path within the integral range to generate 
definite integral infonnation of the change infonnation within the integral range, nomnalizing means (e.g.. nonnalization 
executing section 73 shown in FIG. 36. etc.) for nomnalizing the change information by the definite integral Information 
to output normalized infomnation obtained as the result of the normalization, and Integration executing means (e.g., 
integration executing section 74 shown in FIG. 36, etc.) for integrating the nonmallzed Infomiation to thereby generate, 

25 a key signal. 

[0025] A picture synthesis apparatus according to this invention is directed to a picture synthesis apparatus adapted 
for receiving a first picture consisting of at least foreground picture and background picture, and a second picture 
consisting of at least background pteture to synthesize the foreground picture of the first picture and the background 
picture of the second picture, the apparatus comprising: change detecting means (e.g., change detecting section 10 

30 shown in FIG. 10, 30 or 33, etc.) for detecting change of pixel value between pixels constituting the first picture: key 
signal generating means (e.g., integral value calculating section 40 shown in FIG. 1 0 or 30. or integral value cateulating 
section 70 shown in FIG. 33, etc.) for generating a key signal on the basis of the change of the pixel value outputted 
from the change detecting means; and synthesis means (e.g., mixer 85 shown in FIG, 38 or mixer 96 shown in FIG. 
39. etc.) for synthesizing the foreground picture of the first pteture and the background pteture of the second picture 

35 by using the key signal generated by the key signal generating means. 

[0026] The picture synthesis apparatus according to this Invention further comprises difference detecting means (e. 
g., F-B detecting section 30 shown in FIG. 10 or 30. etc.) for detecting difference between the foreground pteture and 
the background picture of the first picture to output difference infonmation indicating the difference thus detected, where- 
in the change detecting means outputs change information indicating the change of the pixel value, and the key signal 

40 generating means generates the key signal on the basis of the difference information and the change information. 
[0027] It should be noted that it is a matter of course that the foregoing description does not mean that the respective 
means are limited to the above-mentioned components. 

[0028] The theoretical ground (principle) of this invention will now be described. It is to be noted that picture is as- 
sumed to be one-dimensional picture here for the brevity of explanation. 

45 [0029] It is now assumed that pixel value of pixel at position (coordinate) x constituting one-dimensional picture C. 
Moreover, the pteture C Is assumed to consist of foreground (foreground picture) F and background (background pic- 
ture) 8, and pixel values of pixel at the position x constituting the foreground F and the background B are respectively 
designated at F(x) and B(x). It is further assumed that value at the position x of the key signal for extracting the fore- 
ground F from the pteture C (contribution ratio of the foreground F at the position x) is a(x). In this case, it is assumed 

50 that a(x) takes a value within the range from 0 to 1 . and respectively takes 1 and 0 at positions of pixels completely 
constituting the foreground F and the background 8. In addition, at the positions of pixels where the foreground F and 
the background B are mixed. a(x) is assumed to take a value corresponding to the degree of contribution of pixel value 

of the foreground F. . u- u f», 

[0030] In this case, the pixel value C(x) of pixels constituting the picture C is considered to be value in which the 
55 pixel value F(x) of pixels constituting the foreground F and the pixel value B(x) of pixels constituting the background 
B are synthesized at a ratio corresponding to the key signal o(x). Accordingly, the pixel value C(x) can be expressed 
by the following equation (1): 
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C(x) = a(x)F(x) + (1 - a(x))B(x) 



(1) 
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[0031 J Here, the previously described alias takes place because pixel has definite size {dimensions), and motion 
blur takes place in the case where any one of the foreground F and the background 8 is moving, and in the case where 
both are independently moving. Explanation will now be given in a manner classified into four cases: 

[1 1 Case where the foreground F and the background B are stationary, and pixels can be considered to be infinitely 



[2] Case where the foreground F or the background 8 is moving, and pixels can be considered to be infinitely small. 
[3| Case where the foreground F and the background 8 are stationary, and pixels have definite size (dimensions). 
[4] Case where the foreground F or the background 8 is moving, and pixels have definite size (dimensions). 

[1 1 Case where the foreground F and the background 8 are stationary, and pixels are considered to be infinitely small. 

[0032] In this case, since the foreground F and the background 8 are stationary, no motion blur takes place. Moreover, 
since pixels are considered to be infinitely small, no alias also takes place. Accordingly, the key signal a(x) takes any 
one of value of 0 and 1 . and the position x where that value becomes equal to lis in correspondence with the existing 
range of the foreground F. 

[0033] Here, -when the equation (1) is differentiated by x. the equation (2) is obtained. It should be noted that de- 
scription of argument (x) of the function is omitted as the occasion may demand. 



In the above equation. ' (prime) represents differentiation. 

[00341 In the case where ordinary picture is assumed, texture (pixel value) of the foreground F and the background 
B is not uniform, but varies as shown in FIG. 4. However, in the case where the foreground F and the background B 
are locally viewed, e.g., in pixel units, etc., that texture can be considered to be uniform as shown in the FIG. 4 mentioned 
above. Accordingly, from a local point of view, the pixel value of the foreground F and the pixel value of the background 
B can be considered to be constant. As a result, the differential value P and the differential value B* can be both 
considered to be zero (0). From this fact, the equation (2) can be rewritten into the following equation (3): 



[0035] Since the differential value C of the left side of the equation (3) is a value obtained by differentiating pixel 
value C(x) constituting the picture C by x, this differential value C represents change of pixel value between pixels 
constituting the picture C. It is seen from the equation (3) that change C(x) of pixel value between pixels of the picture 
C and differential value a'(x) of the key signal have the proportional relationship in which difference F{x)-B(x) between 
pixel value F(x) of the foreground F and pixel value 8(x) of the background B is caused to be proportional constant. 

[2) Case where the foreground F or the background 8 is moving and pixels can be considered to be infinitely small. 

[0036] In the case where attention is now drawn to picture of moving object, motion blur takes place in the picture 
of the current frame as has been explained with reference to FIG. 3, Namely, the pfeture of the cun^nt frame is the 
picture in which movement of the objert from the frame preceding (earlier) by one relative to the cun-ent frame up to 
the cun-ent frame is reflected. Accordingly, in the case where times of the current frame and the frame preceding by 
one are respectively designated at t2, t^, and pixel value at time t of pixel at a certain position x of the picture C is 
represented by Ct(x), pixel value Ct2(x) of pixel of the position x at the time tg can be represented by time Integration 
of picture Sx(t) whk:h has been passed through the pixel of the position x for a time period from the times t, to tg. 
Namely, the pixel value Ct2(x) can be expressed by the following equation (4): 



small. 




(4) 
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[0037] However, there is no way to find out (recognize) the picture Sx(t) which has been passed through the pixel 
of the position x for the time period from times t^ to tg from the picture of the current frame, or the picture of the frame 
preceding by one. In view of the above, when the picture located at the position x is represented by Stjix) at the time 
tg of the current frame, pictures existing within the range from the positions x^ to Xg are respectively represented by 
pictures St2(x,) to StglXg). In this case, when it is assumed that the pictures StgCx^) to St2{X2) have been passed through 
the pixel located at the position x for the time period from the times t, to tg, whereby pixel value Ct2{x) of picture of the 
position X at the time tg has experienced the influence of the pictures StgCxO to StgCxg), the pixel value Ct2(x) is equal 
to time integration of the picture Sx(l) which has been passed through the pixel of the position x for the time period 
from the times to as shown in FIG. 5B, and can be represented by the spatial integration of the pictures Stglx,) to 
Sl2(X2) existing within the range from the positions to at the time tg. Namely, the pixel value a2{x) can be expressed 
by the following equation (5): 



Ct,(x) = fst^lx)dx ... (5) 

*t 

[0038] Meanwhile, when still picture (picture in which no motion blur takes place) at the time tj is represented by S 
(x) and filter for giving the effect of the motion blur with respect to the still picture is represented by R(x), the right side 
of the equation (5) is equal to convolution integral of S(x) and R(x) as shown in FIG. 6. Accordingly, the equation (5) 
can be rewritten as indicated by the following equation {S). 

25 Ct2(x) = S(x)*R(X) (6) 

In the above equation, * indicates convolution integral. 

[00391 Now. filters which give the effect of the motion blur to the foreground F and the background B are assumed 
to be respectively represented by Rf(x) and Rb(x). In addition, In this embodiment, since the key signal a{x) moves 
along with the foreground F, in the case where the effect of the motion blur is given by the filter Rf(x) with respect to 
the foreground F. it is necessary to give the effect of the motion blur also to the key signal a(x) blur by the same filter 
Rf(x) as shown in FIG. 78. When the effect of the motion blur is given to the foreground F, the background B and the 
key signal a in consideration of the above-mentioned facts to rewrite the equation (1), the following equation (7) is 
provided: 

C = (a*RO(F*Rf) + (1 -a*RO(B*Rb) (7) 
Differentiation of the above equation (7) gives the following equation (8): 

C' = (F*Rf-B*Rb)(a'*RO 
+ (P*Rf . B-Rb)(a*Rf) + B'*Rb (8) 

45 

[0040] If the texture of the picture of the foreground F to which the effect of the motion blur has been given, i.e.. the 
texture of the ptoture obtained by filtering the foreground F by the filter Rf can be considered to be also uniform similarly 
to the case shown in FIG. 4 from a local point view. This similarty applies to the pixel value B of the background B. 
[0041] Accordingly, from a local point of view, the pixel values F'Rf and B*Rb which have been caused to undergo 
so filtering can be considered to be constant As a result, these differential values (F*RO' and (B*Rb)' can be both con- 
sidered to be zero (0). From this fact, the above-mentioned equation (8) can be rewritten as indicated by the following 
equation (9): 

55 C' = (PRf-B*Rb)(a'RO' (9) 

[0042] It is seen from the equation (9) that change C of the pixel value between pixels of the picture C and differential 
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value (a -RO* of the key signal in which the effect of the motion blur is taken into consideration have the proportional 
relationship in which difference F*Rf.B*Rb between F*Rf of the foreground and the pixel value B*Rb of the background 
in which the effect of the motion blur Is similarly taken into consideration is caused to be proportional constant. 

[3] Case where the foreground F and the back ground B are stationary, and pixels have definite size 

[0043] In the case where the size of pixel is considered to be infinitely small, the pixel value smoothly varies with 
respect to the position x as shown in FIG. 9A. In contrast, in the case where pixels have definite size, the pixel value 
vanes m a step form with respect to the posiUon x as shown in FIG. 9B. For this reason, in the case where the signal 
shown in FIG. 9A is merely sampled, alias takes place at the boundary between the foreground F and the background 
8 as previously described. In order to reduce such alias, there is a method, as shown in FIG. 9A. in which, for the 
purpose of calculating the pixel value, the pixel value is caused to undergo filtering by the low-pass filter G(x)'to carry 
out sampling, i.e., value obtained as the result of convolution integral of the pixel value and the low-pass filter G(x) at 
the pixel position is caused to be new pixel value. In view of the above, in the case where picture in which alias is 
reduced by the low-pass filter G(x) in a manner described above is assumed, its pixel value (new pixel value) C(x) can 
be represented by the following equation (1 0) from the above-mentioned equation (1 ): 

. C(x) = (a(x)F(x) + (1 -a(x))B(x))-G(x) (1 o) 

[0044] When the equation (10) Is differentiated by x. the following equation (11) is provided. 

C = ((F-B)a' + (P-B')a+B')*G 
= ((F-B)a')*G + ((F-G)'.(B-G)')o 

+ (B-G)' (11) 

30 [0045] Here, the texture of the pteture obtained as the result of the fad that the pixel value of the foreground F and 
the pixel value of the background B are subjected to filtering by the low-pass filter R can be also considered to be both 
unifomi simiiariy to the case shown in FIG. 4 when they are locally viewed. 

[0046J Accordingly, from a local point of view, the pixel values F*G and B*G which have been caused to undergo 
filtering can be considered to be both constant. As a result, these differential values (F*G)' and (B*G)' can be considered 
35 to be both zero (0). From this fact, the equation (1 1 ) can be rewritten as indicated by the following equation (1 2). 



20 



25 



C = ((F-B)a')-G (12) 

[0047] Moreover, when locally viewed, as explained with reference to FIG. 4. the pixel values of the foreground F 
and the background B can be considered to be both constant. Accordingly, the equation (12) can be rewritten as 
indicated by the following equation (13): 



C = (F-B)(a-G)' (13) 



[0048] It is seen from the above equation (13) that change C of the pixel value between pixels of the picture C and 
the differential value (a*G)' of the key signal for reducing alias have the proportional relationship in whwh difference 
F-B between the pixel value of the foreground F and the pixel value of the background B Is caused to be proportional 
50 constant. 

[4] Case where the foreground F or the background B is moving and pixels have definite size 

[0049] In this case, from the above-described equations (1 ), (7) and (1 0), the pixel value C can be expressed by the 
55 following equation (14): 



C = ((a*RO(F*Rf) + (1 -a-Rf)(B*RO)-G (1 4) 
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[00501 When the equation (14) is differentiated by x, the following equation (15) is provided: 



C = ((F*Rf-B*Rb)(a'*RO 
+ (F'*Rf-B'*Rb)(a*Rf)+8'*Rb)'G 



[0051 ] Here, the texture of the picture obtained by filtering the pixel value of the foreground F by the filter Rf to further 
allow it to undergo filtering by the low-pass filter G can be considered to be also uniform similarly to the case in FIG 
4 when locally viewed. This similarly applies to the pixel value of the background B. 

[0052J Accordingly, from a local point of view, the pixel values F*RfG and 8*Rb-G which have been caused to 
undergo filtering can be considered to be constant As a result, these differential values (F'RfG)'. (B*Rb*G)' can be 
considered to be both zero (0). Further, when locally viewed, as explained with reference to FIG. 4. the pixel value of 
the foreground F and the pixel value of the background B can be considered to be both constant. From this fact, the 
equation (15) can be rewritten as indicated by the following equation (16): 



C = (F*Rf - B*Rb)(a*RrG)' (16) 

20 [0053] It Is seen from the above equation (1 6) that the change C of the pixel value between pixels of the picture C 
and the differential value (a*Rf*G)' of the key signal such that the effect of motion blur is taken into consideration and 
that alias is permitted to be reduced have proportional relationship in which difference F*Rf.B*Rb between the pixel 
value F*Rf and the pixel value B'Rb of the background In which the effect of motion blur Is taken into consideration is 
caused to be the proportional constant. 

25 [00541 It is seen from the above-mentioned equations (3), (9). (1 3) and ( 1 6) that, in all of the above-described cases 
of [1] to [4). the change C of the pixel value between pixels of the picture C and the differential value a' of the key 
signal have the proportional relationship, and the proportional constant is difference F-B between the pixel value of 
the foreground and the pixel value of the background constituting the picture C. Namely, even if motion blur or alias 
does not take place in the pknure C, the proportional relationship between O and o' holds. Moreover, in the case where 

30 motion blur or alias takes place in the picture C, the proportional relationship between C and a* holds also with respect 
to the relationship between the key signal a in which the influence of that motion blur is taken into consideration, or 
the key signal for permitting alias to be reduced and the pixel value C. In any case, the proportional constant becomes 
difference F-B between the pixel value of the foreground and the pixel value of the background constituting the prcture C. 
[00551 Accordingly, a key signal a adapted to allow the proportional relationship to hold is such that in the case where 

35 motion blur takes place in the picture C, it becomes a key signal in which the influence of that motion blur is taken into 
consideration, and that in the case where alias takes place in the pcture C, it becomes a key signal for allowing that 
alias to be reduced. This invention is based on the above-described theoretical ground (principle). 
[0056) FIG. 1 0 shows the configuration of an embodiment of the key signal generating apparatus adapted for gen- 
erating a key signal on the basis of the above-mentioned principle. A picture C for generating key signal is inputted to 

40 a change detecting section 10. Then, the change detecting section 10 cateulates (detects) change C (change infor- 
mation) of pixel value between pixels of the picture C to deliver this change C to a feature point detecting section 20 
and an integral value calculating section 40.. 

[00571 The feature point detecting section 20 detects closed curve for path detemiination and feature point which 
will be described later on the basis of the change C of the pixel value delivered from the change detecting section 10 
45 to deliver the path detemiination closed curve and the feature point to an F-B detecting section 30 and an integral 
value calculating section 40. 

[00581 The F-B detecting section 30 is supplied with the change C of the pixel value from the change detecting 
section 10, and the path determination closed curve and the feature point from the feature point detecting section 20, 
and is supplied with the, same picture C as the picture inputted to the change detecting section 10. Thus, the F-B 
so detecting section 30 calculates (detects) difference F-B (difference infonmation) between the pixel value of the fore- 
ground F and the pixel value of the background B of the picture C to deliver this difference F-B to the integral value 
calculating section 40. 

[0059] The integral value calculating section 40 calculates the skeleton portion which will be described later of key 
signals a con-esponding to the entirety of the picture C on the basis of the change C* of the pixel value delivered from 
55 the change detecting section 1 0, the path determination closed cun/e and the feature point delivered from the feature 
point detecting section 20, and the difference F-B delivered from the F-B detecting section 30 to deliver this skeleton 
portion to a key signal generating section 50. 

[0060] The key signal generating section 50 extracts the foreground F from the picture C by using the skeleton portion 
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delivered from the integral value calculating section 40. and to generate a key signal for synthesizing the extracted 
foreground F with any other picture. Namely, the key signal that the integral value cateulating section 40 outputs is only 
the skeleton portion of the key signals a corresponding to the entirety of the picture C. The key signal generating 
section 50 interpolates portions except for the skeleton portion of the key signal to generate the key signal a corre- 
sponding to the entirety of the picture C. 

[0061] More practical operation of the key signal generating apparatus will now be described with reference to the 
flowchart of FIG. 11 . It is to be noted that picture C is assumed to be picture on the xy plane, i.e.. two-dimensional 
picture in the following description. 

[0062] At step SI , when picture C for generating a key signal Is Inputted, that picture C is delivered to the change 
detecting section 1 0 and the F-B detecting section 30. Then, the change detecting section 1 0 is operative so that when 
picture C is inputted thereto, it calculates change C of the pixel value between pixels of that picture C. Namely, at this 
step SI, with respect to respective pixels constituting the picture C. e.g.. distance on the color space between pixels 
adjacent to each other Is calculated. The distance thus calculated is outputted as change (color change in this case) 
C of the pixel value between pixels of the pteture C. 

[0063] A more practical example of the configuration of the change detecting section 1 0 of FIG. 1 0 is shown In FIG 
12. The picture C is delivered to an RGB decomposing section 11. The RGB decomposing section 11 decomposes 
the picture C into, e.g., components of respective axes directions in RGB (Red. Green, Blue) space, i.e., R-component 
G-component and B-component (hereinafter referred to as R, G. B components as the occasion may demand). These 
R. G. B components are respectively delivered to first (1 -st) order differential value calculating sections 12R 12G and 
12B. 

[0064] The first order differential value calculating sections 12R, 12G, 128 calculates respective first order differential 
values R', G', B' to calculate directions and magnitudes (vectors) R', G*. B* of changes of respective components (I.e., 
gradients of the cun/e assumed to be constituted by pixel values (respective components) of the picture C on the space 
in which axis indicating pixel values vertical to the xy-plane where the picture C exists is added to the xy-plane. Then, 
vectors R', G'. B' of respective changes of the R, G, B components are delivered to a vector synthesis section 13. 
[0065] The vector synthesis section 13 synthesizes the vectors R*. G', B' delivered from the first order differential 
value calculating sections 12R. 12G, 12B. i.e.. synthesizes directions of respective changes of R. G. B components, 
and synthesizes magnitudes of the respective components to calculate change (color change) C of pixel values be- 
tween pixels of the picture C. 

[0066] The operation of the change detecting section 1 0 will now be described with reference to the flowchart of FIG 
13. 

[0067] At step S1-1. the RGB decomposing section 11 decomposes the pteture C into components of respective 
axes directions of R G B space. I.e.. R, G, B comppnents. It is to be noted that the pteture C may be decomposed into 
Y, U, V components in color space except for the RGB space, e.g.. YUV space. Namely, the color space employed is 
not partteularfy limited. In addition to decomposing the prcture C into components of respective axes directions of the 
color space, luminance components may be extracted therefrom to carry out processing of the succeeding stage only 
with respect to the luminance components. Then, the R. G, B components obtained at the step SI -1 are respectively 
delivered from the RGB decomposing section 11 to the first order differential value calculating sections 12R, 12G. 12B. 
[0068] At step SI -2. the first order differential value calculating sections 12R. 12G, 12B is operative so that when 
the R, G. B components are delivered thereto, they calculate the first order differential values R', G'. B' of these R, G, 
B components. 

[0069] It is to be noted that, in this embodiment, first order differential values of respective components are determined 
by canying out. e.g. . convolution of respective components and sobel operator, etc. Namely, at the first order differential 
value calculating sections 12R, 12G, 12B, values obtained by canylng out convolution of x or y direction components 
of the respective R, G, B components and sobel operator SO, or SOy shown in FIG. 14 are caused to be respectively 
first order differential values f^, fy in the x or y direction. 

[0070] First order differential values of the R, G, B components determined In a manner described above, i.e. , chang- 
es of the R. G, B components are si^ that their magnitudes (gradients) are maximum at respective pixels. Accordingly, 
the directions of changes of respective components are in correspondence with the directions in which the edge in- 
tensity is maximum. Namely, at this step SI -2, as shown in FIG. 15, changes of the R, G, B components with respect 
to the direction caused to be in correspondence with the direction in which the edge intensity Is maximum of any 
(arbitrary) directions of 360 degrees with respect to the remarked pixel (tiie portion to which slanting lines are attached 
in FIG. 15) are detennined. 

[0071] Then, first order differential values R', G\ B* of R. G, B components respectively detemiined at the first order 
differential value calculating sections 12R. 12G, 12B (these values are vectors because they have magnitude and 
direction as described above) are delivered to the vector synthesis section 13. 

[0072] At step SI -3. the vector synthesis section 13 synthesizes first order differential values R'. G', B' to calculate 
change (color change) C* of the pixel value between pixels of tiie picture C. Namely, if first order differential values R. 
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G\ B' are respectively represented by (r,. ry). (g,. g^), (b,. by), and C is represented by (c„ Cy) (a and b of (a b) 
respectively indicate x-componenl and y^omponenl). At this step SI -3. x component c^ and y component c^ of change 
C of the pixel value are respectively detemiined by the expression r,+g,4-b, and ry^-gy^-by. Then, change Cof the pixel 
value detemfiined in a manner as stated above at the change detecting section 10 is delivered to the feature point 
detecting section 20 and the integral value calculating section 40. Thus, the processing at the change detecting section 
10 is completed. 

[0073] At step S2 (FIG. 11), the feature point detecting section 20 is operative so that when the change C of the 
pixel value is delivered thereto from the change detecting section 10, it detects the path determination closed curve 
and the feature point from the change C of that pixel value. 

[0074] A more practical example of the configuration of the feature point detecting section 20 is shown in FIG 1 6 
A boundary (border) detecting section 21 is supplied with the change C" of the pixel value from the change detecting 
section 10. Thus, the boundary detecting section 21 detects, from the change C of the pixel value, boundary between 
the foreground (object) and the background B which constitute the picture C. Namely, the boundary detecting section 
21 IS operative, as shown in FIG. 17A, for example, to detect the points (pixel) where the change C of the pixel value 
IS well to detect, as the boundary between the foreground F and the background B (boundary of the object in the picture 
C), closed curve constituted by connecting such points (since this closed curve is used for determining integral path 
which will be described later, it will be called hereinafter path detenmination closed curve as the occasion may demand). 
Then, the path determination closed curve detected at the boundary detecting section 21 is delivered to the feature 
point determining section 22. 

[0075] The feature point detennining section 22 determines, as the feature point, predetermined points on the path 
detemiination closed curve delivered from the boundary detecting section 21 as shown in FIG. 17B. for example. 
Namely, the feature point detemnining section 22 detects, e.g., the points where curvature Is high of points constituting 
the path detemnination closed curve to determine those points as the feature point. 

[0076] Moreover, the feature point detemriining section 22 is operative, as shown in FIG. 1 7C, for example, to deter- 
mine the dosed curve surrounding the path detenninatlon closed curve (the portion indicated by dotted lines In FIG. 
17C). and divides the area of the background B sun-ounded by that closed curve and the path detemiination closed 
curve into several areas where the change C* of that pixel value is relatively constant (hereinafter refen-ed to as back- 
ground constant area as the occasion may demand). Further, the feature point detennining section 22 partitions the 
path determination closed curve in the background constant area units so that divided very small areas are provided. 
30 In addition, the feature point determining section 22 detects middle points in respective very small areas to determine 
those points as the feature point. 

[0077] The operation of the feature point detecting section 20 will now be described with reference to the flowchart 
of FIG. 18. 

[0078] At step S2-1 . the boundary detecting section 21 is operative so that when the change C of the pixel value is 
35 delivered thereto from the change detecting section 1 0, it detects the points where the change a of that pixel value is 
great to connect those points to thereby detect a path detemnination closed curve. Then, the path determination dosed 
curve is delivered to the feature point detennining section 22. 

[0079] At step S2-2. the feature point determining section 22 Is operative so that when the path determination closed 
curve is delivered thereto from the boundary detecting section 21, it detennines, as the feature point, predetermined 
points on the path determination closed curve. Then, the path detemiination closed curve and the feature points re- 
spectively obtained at the boundary detecting section 21 and the feature point detemiining section 22 are delivered to 
the F-B detecting section 30 and the integral value calculating section 40. Thus, the processing at this feature point 
detecting section 20 is completed. 

[0080] At step S3 (FIG. 11). the F-B detecting section 30 is operative so that when the path detemiination closed 
^5 curve and the feature points are delivered thereto from the feature point detecting section 20, it calculates difference 
F-B between the pixel value of the foreground F and the pixel value of the background B which constitute the picture C. 
[0081] A more practical example of the configuration of the F-B detecting section 30 is shown in FIG. 1 9. The very 
small area detecting section 31 is supplied, in addition to the picture C (picture value), with the feature points and the 
path detennination closed curve from the feature point detecting section 20. Thus, the very small area detecting section 
so 31 detects the very small area including the feature points as shown in FIG. 20A, for example. Namely, the very small 
area detecting section 31 divides the path detennination closed curve into very small line segments at points equi- 
distant from the feature point as shown in FIG. 208 in the state where a portion of FIG. 20A is enlarged to detect, as 
the very small area (the portion to whtoh slanting lines are attached in FIG. 20B). the range spaced by a predetermined 
distance (e.g., distance corresponding to four pixels, etc.) toward the side of the foreground F or the background B 
from that line segment. Moreover, the very small area detecting section 31 extracts, with respective to respective very 
small areas, pixels constituting the foreground F included within those areas (pixels existing at the foi-eground F side 
from the path determination closed curve, i.e.. the portion to which slanting lines going down to the left are attached 
in FIG. 20B) to deliver pixel value of those pixels (hereinafter referred to as F boundary pixel value as the occasion 
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may demand) to a color distribution fomiing circuit 32F. and extracts pixels constituting the background 8 (pixels existing 
at the background B side from the path determination closed curve, i.e., the portion to which slanting lines going down 
to the right are attached in FIG. 208) to deliver pixel value of those pixels (hereinafter referred to as B boundary pixel 
value as the occasion may demand) to a color distribution forming section 328. 

[00821 The color distribution fonning sections 32F. 328 respectively detemiines color distributions on the color space 
e.g., RGB space, etc. of the F boundary pixel values and the 8 boundary pixel values every very small areas delivered 
from the very small area detecting section 31 to respectively deliver them to center of gravity detecting sections 33F 
338. The center of gravity detecting sections 33F. 338 respectively detemiine centers of gravity of the color distributions 
of the F boundary pixel values and the B boundary pixel values delivered from the color distribution forming sections 
32F, 32B to considers respective centers of gravity as pixel values of the foreground F and the background 8 of pixels 
in the gray area to deliver them to a difference calculating section 34. 

{0083] The difference calculating section 34 is operative, as shown in FIG. 21 . for example, to subtract, from the 
center of gravity of the color distribution of the F boundary pixel values delivered from the center of gravity detecting 
section 33F. the center of gravity of the color distribution of the B boundary pixel values delivered from the center of 
gravity detecting section 338 to output the subtraction result as difference F-8 between the foreground F and the 
background 8 (different of pixel value between the pixel value of the foreground F and pixel value of the background 
8, and this difference is thus considered to be vector similarly to C) in the very small area (or at feature point included 
within that very small area). 

[0084] The operation of the F-B detecting section 30 will now be described with reference to the flowchart of FIG. 22. 
[0085J At step S3-1 , the very small area detecting section 31 detects the very small area, which has been explained 
with reference to FIG. 20. on the basis of the feature points and the path detemiination closed curve delivered from 
the feature point detecting section 20 to further extract F boundary pixel values and 8 boundary pixel values in that 
very small area to respectively delh^er the F boundary pixel value and the 8 boundary pixel value to the color distribution 
forming sections 32F, 328. 

25 [0086] At step S3-2, the color distribution forming sections 32F, 328 respectively determines color distributions on 
the RGB space of the F boundary pixel values and the 8 boundary pixel values in a certain very small area to deliver 
them to the center of gravity detecting sections 33F, 338. 

[0087] At step S3-3, the center of gravity detecting sections 33F. 338 respectively calculates (detect) centers of 
gravity of color distributions of the F boundary pixel values and the 8 boundary pixel values to deliver them to the 
30 difference calculating section 34. 

[0088] At step S3-4, the difference calculating section 34 detenmines difference value between the center of gravity 
of the color distribution of the F boundary pixel values delivered from the center of gravity detecting section 33F and 
the center of gravity of the color distributton of the B boundary pixel values delivered from the center of gravity detecting 
section 338 to output this difference value as difference F-B' between the foreground F and the background B at the 
35 feature point included within the very small area. 

[0089] At step S3-5, after the difference F-B between the foreground F and the background at the feature point 
included within a certain very small area is obtained in a manner as stated above, whether or not the difference F-B 
is determined with respect to alt very small areas detected at the very small area detecting section 31 (ail feature points 
detected at the feature point detecting section 20) is judged. In the case where it is judged at the step S3-5 that the 
difference F-8 has not yet been detenmined with respect to all very small areas, the processing operation returns to 
the step S3-2 to cany out the processing at step S3-2 and steps subsequent thereto with respect to the very small 
areas where the difference F-8 has not been determined. On the other hand, in the case where it is judged at the step 
S3-5 that the difference has been determined with respect to all the very small areas, the processing at the F-B detecting 
section 30 is completed. Then, the differences F-8 which have been detenmined with respect to all the very small areas 
are delivered to the integral value calculating section 40. 

[0090] It is to be noted that the direction of the difference F-B (vector) between the foreground F and the background 
8 obtained in a manner as described above is in correspondence with (substantially in con-espondence with) the di- 
rection in which the edge intensity becomes maximum, and the direction of F-8 and the direction of C are in corre- 
spondence with each other (substantially in correspondence with each other). 

[0091] Further, while the F-B detecting section 30 shown In FIG. 1 9 determines the very small area on the basis of 
the path detemiination closed curve (the boundary between the foreground F and the background B) obtained from 
the change C of the pixel value to detemriine the difference F-B from the very small area, the difference F-B may be 
detemiined. e.g., in a manner stated below at the F-B detecting section 30. In more practical sense, e.g., not only the 
frame of the picture C but also frames constituting moving pk:ture before and after that frame is inputted to the F-B 
detecting section 30 in a time series manner. Then, the F-8 detecting section 30 stores the frame inputted at present 
until the subsequent frame is inputted. In the case where picture C for generating a key signal is Inputted, frame of 
that picture C and the stored frame preceding (eariler) by one are caused to undergo the so-called block matching to 
detect the boundary between the foreground F and the background B on the basis of the above-mentioned result and 
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the result of the block matching preceding thereto as occasion demands. Then, the F-B detecting section 30 determines 
feature points on the path determination closed curve estimated from the boundary by using the boundary to detect 
the very small area including respective feature points similariy to the case at the steps S3-1, S3-2 of FIG 22 to 
recognize respective distributions of the foreground F and the background 8 whk:h constitute the picture C every 
respective very small areas to detemnine centers of gravity of respective color distributions of the foreground F and 
the background 8 similarly to the case at the steps S3.3. S3-4 of FIG. 22 at times subsequent thereto to further deter- 
mine direction of the position of the center of gravity of the color distribution of the background F with respect to the 
position of the center of gravity of the color distribution of the background B. and distance on the color space therebe- 
tween. i.e., the difference F-B between the fore is unnecessary to deliver the path determination dosed curve and the 
feature points from the feature point detecting section 20 to the F-B detecting section 30. It Is to be noted that the detail 
of the method of detecting the boundary between the foreground F and the background 8 by using block matching as 
described above is disclosed, e.g.. in the Japanese Patent Application No. 164792/1995, etc. 
[0092] Moreover, while the method of detecting the boundary between the foreground F and the background B is 
not limited to the method using change C of the pixel value and the method using block matching as described above 
detection of the boundary between the foreground F and the background B may be carried out by instruction from 
users as described later in FIG. 30, for example. Further, the boundary between the foreground F and the background 
8 may be detected by employing in combination, e.g.. the method by instruction from users and the method using 
change C of pixel value. Furthermore, the boundary between the foreground F and the background 8 may be detected 
by employing in combination, e.g. . the method by instruction from users and the method using block matching. Namely, 
e.g., an approach may be employed to detect the boundary between the foreground F and the background B by in- 
structlon from users with respect to the first frame, and to detect the boundary between the foreground F and the 
background B by block matching with respect to frames subsequent thereto. 

[0093J Further, the method of detecting the difference F-B is not limited to the above-described method, but may 
employ other methods. 

[0094] Moreover, while after the processing of the step S2. the processing of the step S3 is carried out in the flowchart 
shown in FIG. 11. in the case where there is no necessity of delivering the path determination closed curve and the 
feature points with respect to the F-B detecting section 30 from the feature point detecting section 20, the processing 
of the step S2 and the step S3 may be earned out in an order opposite to the above, or may be carried out at the same 
time (in parallel). The difference F-B detennlned in a manner as described above at the step S3 (FIG. 11) is delivered 
from the F-B detecting section 30 to the integral value calciilating section 40. 

[0095] At step S4. the integral value calculating section 40 normalizes change C of pixel value delivered from the 
change detecting section 10 by absolute value !F-B! of difference F-B delivered from the F-B detecting section 30. 
[0096] At step S5, the Integral value calculating section 40 Integrates the nomnalized value C'/IF-BI along, e.g., the 
direction in which change C* of pixel value Is caksulated (detected) to thereby calculate the skeleton portion of the key 
signal a. 

[0097] In more practical sense, since the change C of pixel value between pixels of the picture C and the differential 
value a' of the key signal have the proportional relationship therebetween as described above, and the proportional 
constant is difference F-B between the foreground and the background which constitute the picture C. i.e., the expres- 
sion C'=(F-B) a* holds, the key signal a(p) can be determined by the following equation (17). 

[0098] In the above equation. a(p) means a key signal with respect to pixel at the position (point) p(x. y) constituting 
the picture C, and F(p) and B(p) respectively means pixel values of pixels at the position p(x, y) of the foreground F 
and the background B which constitute the picture C, Further, C(p) means pixel value of pixel at the position p(x. y) 
constituting the picture C. It is to be noted that description of the argument (p) of the function will be hereinafter omitted 
as the occasion may demand. 

[0099] Accordingly, at the above-descn*bed step S4, C'/tF-Bl constituting the right side of the equation (17), i.e., 
nomfialized value obtained by nonmalizing the change C of the pixel value by the absolute value IF-b! of difference is 
detemiined. At this step S5. integration at the right side of the equation (1 7), i.e.. integration of the normalized value 
C*/IF-BI along, e.g.. the direction in which change C of the pixel value is detected is carried out. Value obtained as the 
result of integrating value obtained by normalizing change C of the pixel value by the absolute value IF-Bl of difference 
without integrating the change C of the pixel value as it is. i.e., the key signal a takes value within the range from 0 to 1 . 
[0100] The integration canied out at the step S5 will now be described in more detail. When space In which the z- 
axis perpendicular to the xy plane where the picture C exists is provided Is assumed to be on the xy-plane is assumed 
as shown in FIG. 23, for example, to take pixel value in the z-axis direction, plane (curved surface) V consisting of pixel 
values is formed within that space. It should be noted that if the picture C is color picture (picture image), there are 
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three components such as RGB or YUV. etc. as described above, but the pixel value is assumed to consist of one 
component for the brevity of explanation. 

(0101J In this case, gradient at an intersecting point of a straight line passing through a certain point p on the picture 
C and perpendicular to the xy-plane and the plane V, i.e.. change of pixel value C(p) of pixel located at the position p 
can be represented as vector in the nnaximum inclination direction when the plane V is viewed from that point C(p) 
(vector obtained by calculating gradient of the plane V at the point C(p)). 

[01021 Here, the maximum inclination direction is the direction in which in the case where a ball having size assumed 
to be infinitely small is placed at the point C(p), that ball rolls. 

[0103] Let now suppose that in the case where a ball is placed at the point C(p) and that ball falls (rolls) down to the 
point C(Po). the locus (path) that the ball depicts on the plane V is U. In this instance, the integration of the equation 
(17) means the integration along the path (hereinafter refen-ed to as the maximum inclination direction path as the 
occasion may demand), and the equation (17) means that the integration expressed thereby is carried out whereby 
key signal a(p) with respect to the pixel located at the position p can be detemiined. 

[01041 Accordingly, if the key signal a(Po) with respect to the pixel located at the position Po can be recognized it Is 
possible to detennine the key signal o(p) with respect to the pixel located at the position p in accordance with the 
following equation (18) from the equation (17): 
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■ = (18) 

[0105] In the above equation, integral symbol in which U is attached at the upper right means the Integration along 
the maximum inclination direction path U. ^ » 

[01 06] At step S5, e.g.. calculation of the right side of the equation (1 8) is earned out, or the like, whereby the key 
signal a (the skeleton portion of the key signal) is determined. 
2S [01071 It is to be noted that it is not necessarily required to carry out the integration of the right side of the equation 
(18) along the maximum inclination direction path U. Namely, integration along path (FIG. 23) which is not the 
maximum inclination direction path U as the path from the point C(p) up to the point C(Po) may be carried out Namely 
smce the integration of the right side of the equation (18) means the integration of gradients at respective points orJ 
the maximum inclination direction path U , in the case where integration is carried out along other path U- the component 
in the direction of the maximum inclination direction path U of gradients at respective points on the path is integrated 
(Accordingly, this integration can be also considered to be eventually the same as the integration along the maximum 
inclination dlrectk>n path U). thereby making it possible to determine the same key signal a as that in the case where 
integration is carried out along the maximum inclination direction path U. 

[0108] It is to be noted that since in the case where integration is carried out along paths except for the maximum 
inclination direction path the quantity of operations is increased to more degree as compared to the case where 
integration is carried out along the maximum inclination direction path, it is preferable to carry out integration along the 
maximum inclination direction path U as far as possible. 

[0109J An example of the configuration of the integral value cateulating section 40 of FIG. 10 for carrying out a 
processing as described above is shown in FIG. 24. The path detemiination closed curve and the feature points from 
the feature point detecting section 20 are delivered to an integral path determining section 41 , and change C of the 
pixel value from the change detecting section 10 and difference F-B from the F-B detecting section 30 are delivered 
to a normalization executing section 42. 

[0110] The integral path determining section 41 detemiines integral path for carrying out integration of the equation 
{ 1 8) on the basis of the path determination closed curve and the feature points delivered from the feature point detecting 
section 20. Namely, as shown in FIG. 25A, the integral path determining section 41 determines, as the integral path 
with- respect to con-esponding feature point, a line in a normal direction of the path determination closed curve and 
passing through the corresponding feature point. The integral path with respect to the feature point determined in a 
manner as described above is delivered to the normalization executihg section 42 and the integration executing section 
43. It is to be noted that the Integral path detennlned in this way Is the path substantially along the maximum inclination 
50 direction path U. 

[0111] The nonnalization executing section 42 detennines normalized value C'/IF-BI (normalized information) ob- 
tained by normalizing change C of pixel value of pixel existing on the integral path delivered from the integral path 
determining section 41 by absolute value IF-BI of difference F-B to deliver it to the integration executing section 43. 
The Integration executing section 43 carries out integration of the normalized value C'/IF-BI along the integral path 
S5 from the integral path determining section 41 in accordance with the equation (18) to determine key signal a on the 
integral path. The key signal on the integral path is delivered to the closed cun/e generating section 44. 
[0112] The closed curve generating section 44 generates a PI closed curve and a P2 closed curve which will be 
described later from the key signal a delivered from the integration executing section 43. 
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[01 1 3] Namely, while the integration in accordance with the equation ( 1 8) is carried out along the integral path shown 
in FIG. 25A at the integration executing section 43. whereby the key signal a on the integral path can be obtained that 
key signal a becomes equal to 0 (zero) at the boundary between the gray area and the background B and becomes 
equal to 1 at the boundary between the gray area and the foreground R Accordingly, as shown in FIG 25A the closed 
curve generating section 44 determines points where key signals a on respective integral paths become equal to 0 
(points where a is equal to 0, which will be referred to as p, as the occasion may demand hereinafter) and determines 
points where key signals a become equal to 1 (points where a becomes equal to 1. which will be referred to as p, as 
the occasion may demand hereinafter), thus to connect the points p, on respective integral paths to each other and 
to connect the points on the respective integral path to each other Then, as shown In FIG. 25B. the closed curve 
generating section 44 outputs, as P1 closed curve, the closed cun/e obtained by connecting the points p, to each other 
and outputs, as P2 closed curve, the closed curve obtained by connecting the points pj to each other. The key signal 
' a obtained at the integration executing section 43 and the PI closed curve and the P2 closed curve obtained at the 
closed curve generating section 44 are delivered to the key signal generating section 50 (FIG. 10) 
[01 1 4] The operation of the integral value cak:ulating section 40 will now be described with reference to the flowchart 
of FIG. 26. 

[01151 At step S4.1 . the integral path determining section 41 allows any one of feature points delwered from the 
feature point detecting section 20 to be remarked feature point to determine a straight line passing through the remarked 
feature point and in normal direction of the path detemnlnation closed curve delivered from feature point delecting 
section 20 as the integral path with respect to the remarked feature point. The integral path with respect to the remarked 
feature point is delivered to the normalization executing section 42 and the integraUon executing section 43 
(01161 At step S4-2, the normalization executing section 42 divides the change C of the pixel value of the pixel 
existing on the integral path delivered from the integral path detemiining section 41 by absolute value !F-Bi of the 
difference which is the difference F-B delivered from the F-B detecting section 30 and is the difference determined 
from the very small area (FIG, 20) including the remarked feature point to deliver the normalized value C'/l F-BI obtained 
as the result thereof to the integration executing section 43. 

[01171 At step S4-3, the integration executing section 43 integrates the nomialized value C'/!F-a' In accordance with 
the equation (18) along the Integral path from the integral path determining section 41 by using the integral path and 
the normalized value CYIF-BI with respect to the pixel existing on the integral path respectively delivered from the 
integral path detemiining path and the nonmalization executing section 42 to calculate a key signal a on that integral 
path. This key signal a is delivered to the closed curve generating section 44. In this case, at the integration executing 
section 43. e.g., the remarked feature point is regarded as the point where a becomes equal to 0.5. and the Integration 
of the equation (1 8) is carried out with the remarked feature point being as the initial point. Accordingly, in this example, 
the Integration of the equation (1 8) is carried out in the state where a(Po) is caused to be 0.5. 
[01181 At step S4-4, the integral path detemiining section 41 judges whether or not all of feature points delivered 
from the feature point detecting section 20 have been caused to be the remarked feature point. In the case where the 
integral path detemiining section 41 judges that all of the feature points from the feature point detecting section 20 
have not yet been caused to be the remarked feature point, its processing operation returns to the step S4-1. As a 
result, at the integral path determining section 41. feature point whteh is not yet caused to be the remarked feature 
point is newly caused to be the remarked feature point. At times subsequent thereto, processing as described above 
will be repeated. 

[01191 On the other hand, in the case where the integral path detemiining section 41 has judged at the step S4-4 
that all the feature points have been already to be caused to be the remarKed feature point. I.e.. in the case where the 
Integration of the equation (1 8) along all of Integral paths passing through respective feature points delivered from the 
feature point detecting section 20 is carried out. whereby key signals a on those integral paths (hereinafter referred to 
as the skeleton portion as the occasion may demand), the processing proceeds to step S4-5. 
[01 201 At the step S4-5, the closed curve generating section 44 detects the point p^ (the point of a 0) and the point 
Pa (the point of a = 1) from the key signals o on respective integral paths delivered from the integration executing 
section 43 to connect the points p^ to each other and to connect the point pg to each other to thereby generate a PI 
closed curve and a P2 closed curve. Thus, the processing at the Integral value cateulating section 40 is completed. 
[01211 It is to be noted that, at the integraUon executing section 43. the integration along the integral path delivered " 
from the integral path determining section 41 is carried out with respect to. e.g., segment (line segment) (hereinafter 
referred to as path segment as the occasion may demand) between the point where the key signal a becomes equal 
to 0 (the point p^) and the point where it becomes equal to 1 (the point p^) of that integral path, i.e., the integral path 
within the gray area. This is because since key signals a in areas except for the gray area take any one of values 0 
and 1 , it is unnecessary to determine a with respect to such areas in accordance with the equation (1 8). Accordingly, 
only the key signal a on the path segment is calculated at the integration executing section 43. and the above-described 
skeleton portion means the path segments in all integral paths delivered from the Integral path detemiining section 41 . 
[01221 The skeleton portion of the key signal, and the PI closed curve and the P2 closed curve obtained at the 
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mtegral value calculating section 40 in a manner as described above are delivered to the key signal generating section 

[01231 At step S6 (FIG. 11), the key signal generating section 50 canies out interpolation with respect to the skeleton 
portion of the key signal to generate a key signal a corresponding to the entirety of the picture C. Thus, the processing 

5 at the key signal generating section 50 is connpleted. 

[0124] A more practical example of the configuration of the key signal generating section 50 is shown in FIG. 27. An 
interpolating section 5 1 for inside of the gray area is supplied with the skeleton portion of the key signal from the Integral 
value calculating section 40 and key signals on the PI closed curve and the P2 closed curve from 0/1 allocating section 
52 which will be described later. Thus, the interpolating section 51 for inside of the gray area carries out interpolation 

10 of the key signal in the area except for the skeleton portion, the PI closed curve and the P2 closed curve of the gray 
area shown in FIG. 28A in such a manner that slanting lines are attached (the area surrounded by the skeleton portion 
the PI closed curve and the P2 closed curve). 

[01 25] Namely. FIG. 28A shows xya space in which picture C is disposed on the xy-plane and axis (a-axis) perpen- 
dicular to that xy-plane is taken. It is now assumed that key signals at respective points on the picture C are plotted in 
the a-axis direction within this xya space. Since the key signals delivered to the interpolating section 51 for the inside 
of the gray area are the skeleton portion of the key signal (key signal on the path segment), key signals on the PI 
closed curve (since the PI closed curve is the curve in which points p^ where a is equal to 0 are connected, values of 
key signals thereat are 0 (zero)), and key signals on the P2 closed curve (since the P2 closed curve is the curve in 
whteh points pj where a is equal to 1 are connected, values of key signals thereat are 1), points corresponding to the 
skeleton portion of the key signal are plotted on the path segment, and points of a = 0 and a = 1 are respectively plotted 
on the P1 closed curve and the P2 closed curve. Thus, a frame as indicated by thick line In FIG. 28A (hereinafter 
refen-ed to as the skeleton frame as the occasion may demand) is formed on the xya space. 
[01261 Then, the interpolating section 51 for the inside of the gray area generates cun^ed surface Q for interpolating 
the inside of the skeleton frame to allow a that this curved surface Q provides (designates) to be key signals a at 
respective points within the area formed when the curved surface Q is projected onto the xy-plane (portions to which 
slanting lines are attached in FIG. 28B which will be referred to as patch areas as the occasion may demand). 
[0127] In this case, the interpolating section 51 for the inside of the gray area cames out. as shown in FIG. 288. 
interpolation of key signals as described above with respect to the areas sun-ounded by the PI and P2 closed curves 
and the path segment, i.e., all of the patch areas, 

[0128] The skeleton portion of the key signal, the key signals (a = 0) on the PI closed curve, the key signals (o = 1 ) 
on the P2 closed curve, and the key signals in the patch area obtained by interpolation are delivered from the interpo- 
lating section 51 for the Inside of the gray area to a synthesis section 54. 

[0129] The 0/1 allocating section 52 is supplied with the P1 closed curve and the P2 closed curve. The 0/1 allocating 
section 52 respectively allocates 0 and 1 as key signals a on the PI closed curve and the P2 closed curve to deliver 
them to the interpolating section 5 1 for the inside of the gray area. Moreover, an interpolating section 53 for area except 
for the gray area is supplied with the P1 closed curve and the P2 closed curve. As shown in FIG. 28B, the interpolating 
section 53 for area except for the gray area interpolates 0 as key signals at respective points within the area put between 
the picture frame of the picture C and the PI closed curve (the area considered to be completely background B, which 
will be referred to as aO area as the occasion may demand hereinafter), and Interpolates 1 as key signals at respective 
points within the area surrounded by the P2 closed curve (the area considered to be completely the foreground F, 
which will be refen^ed to as a1 area as the occasion may demand hereinafter), thus to deliver them to the synthesis 
section 54. 

(01 30] The synthesis section 54 synthesizes (an-anges (allocates) at conresponding positions within the picture frame 
of the picture C) the skeleton portion of the key signal, the key signals on the PI closed curve, the key signals on the 
P2 closed curve, and the key signals of the patch area obtained by interpolation from the interpolating section 61 for 
the inside of the gray area, and the key signals of the aO area and the key signals of the a1 area from the interpolating 
section 53 for area except for the gray area, thereby to generate a key signal corresponding to the entirety of the picture 
C. 

[01311 The operation of the key signal generating section 50 will now be described with reference to the flowchart 
so of FIG. 29. 

[0132] At step S5-1 , the 0/1 allocating section 52 respectively allocates 0, 1 to the PI closed curve and the P2 closed 
curve to generate key signals on the Pi closed curve and key signals on the P2 closed curve. The key signals on the 
PI closed curve and the key signals on the P2 closed curve are delivered to the interpolating section 51 for the inside 
of the gray area. 

[01331 At step S5-2. the Interpolating section 51 for the Inside of the gray area is operative so that when the key 
signals on the P1 closed curve and the key signals on the P2 closed curve are delivered thereto from the 0/1 allocating 
section 52. it fomis the skeleton frame as has been explained with reference to FIG. 28A to further generate curved 
surface (Interpolation curved surface) Q for Interpolating the inside of the skeleton portion. 
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[0134] At step S5-3, the interpolating section 51 for the inside of the gray area generates key signals of the patch 
areas on the basis of the curved surface Q to deliver thenn to the synthesis section 54. 

[01 3S] At step S5-4. the interpolating section 53 for area except for the gray area interpolates 0 as key signals at 
respective points within the aO area, and interpolates 1 as key signals at respective points within the a1 area thus to 
deliver them to the synthesis section 54. 

(0136) At step S5-5. the synthesis section 54 synthesizes the skeleton portion of the key signal, the key signals on 
the P1 Closed curve, the key signals on the P2 closed area and the key signals of the patch areas obtained by inter- 
polation delivered from the interpolating section for the inside of the gray area, and the key signals of the aO area and 
the key signals of the a1 area delivered from the interpolating section 53 for area except for the gray area to generate 
a key signal corresponding to the entirety of the picture C. Then, the synthesis section 54 converts this key signal 
(continuous key signal) into key signals of pixel units by implementing, e.g.. filtering, etc, thereto as previously described 
to output these key signals as a key signal for extracting the foreground F from the picture, and allowing the extracted 
foreground F to undergo synthesis (keying) with respect any other picture. Thus, the processing at the key siqnat 
generating section 50 is completed. 

[01 37) It is to be noted that the processing of the step S5-4 may be carried out prior to the processing of the steps 
S5-1 to S5-3, or may be carried out at the same time (in parallel). 

[0138] Further, while the key signals of the patch areas are obtained from the curved surface Q for interpolating the 
skeleton frame in the above-described case, the key signals of the patch areas may be determined, in addition to the 
above, e.g.. in a manner as described below. Namely, an approach may be employed to determine key signals at points 
adjacent to the path segment, the PI closed curve and the P2 closed curve within the gray area on the basis of the 
skeleton portion of the key signal, the key signals on the P1 closed curve and the key signals on the P2 closed curve 
which have been already obtained to sequentially, at times subsequent thereto, key signals at points adjacent to points 
where the key signals have been determined on the basis of already determined key signals. 
[01391 Moreover, while such an approach is employed in the above-described case to determine key signals of the 
patch areas from the curved surface Q for interpolating the skeleton frame fonned by plotting points corresponding to 
the skeleton portion of the key signal onto the path segment, and respectively plotting respective points of a = 0 a = 
1 onto the P1 closed curve and the P2 closed curve, the key signals of the patch areas may be determined, in addition 
to the above-mentioned approach, e.g., from a curved surface for.lnterpolating frame obtained by further plotting points 
of a = 0.5 as well onto the path determination closed curve. 

[0140] While the path determination closed curve, i.e.. boundary between the foreground F and the background B 
Is detemiined from change C of the pixel value in the embodiment of FIG. 10. the path determination closed cun/e 
may be determined on the basts of Instruction from users of the apparatus. 

[01411 A more practical example of the configuration of a key signal generating apparatus for calculating a path 
determination closed curve on the basis of instruction from a user is shown in FIG. 30. It is to be noted that the same 
reference numerals are respectively attached to portions con-esponding to the key signal generating apparatus shown 
in FIG. 10, and the explanation will be omitted as the occasion may demand in the following description. 
[01421 Namely, at this key signal generating apparatus, a feature point detecting section 60 is provided in place of 
the feature point detecting section 20. and an operation section 1 is newly provided. In addition, other portions of this 
key signal generating apparatus are respectively constituted in the same manner as in the key signal generating ap- 
40 paratus shown in FIG. 1 0. 

[0143] The operation section 1 Is composed of, e.g.. tablet and pen or mouse, etc., and is operated when a user 
designates the boundary between the foreground F and the background B while looking at picture C displayed on 
display (not shown). In addition, an operation signal corresponding to that operation is delivered to the feature point 
detecting section 60. 

[0144] The feature point detecting section 60 detects the path detennlnatlon closed cun^e and the feature points on 
the basis of the operation signal from the operation section 1. A more practteal example of the configuration of the 
feature point detecting section 60 is shown in FIG. 31 . It Is to be noted that, In the feature point detecting section 60 
shown in FIG. 31 , the same reference numerals are respectively anached with respect to portions corresponding to 
the feature point detecting section 20 shown in FIG. 16 and their explanation will be omitted as the occasion may 
demand in the following description. Namely, a line thinning section 61 is provided, in place of the boundary detecting 
section 21 , at the feature point detecting section 60, and other portions of the feature point detecting section 60 are 
constituted in the same manner as in the feature point detecting section 20. 

[0145] The line thinning section 61 recognizes line designated by user (hereinafter referred to as designation line as 
the occasion may demand), which corresponds to the boundary between the foreground F and the background 8, on 
the basis of the operation signal delivered from the operation section 1 to carry out line thinning to thereby detect a 
path determination closed cu rye. For example, the designation line is displayed on a display where picture C is displayed 
in a manner overiaid upon the picture C, and is line having width of one pixel or more as shown in FIG. 32A. In view 
of this, the line thinning section 61 determines, as shown in FIG. 32B, for example, in order to detemnine path deter- 
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mlnation dosed curve as an ideal line having no width from such designation line having width, a line (curve) which 
minimizes square error of distances from the centers of pixels constituting the designation line (portions Indicated by 
mark x in FIG. 328) by the least square (method) to allow this line to be the path determination closed curve. 
[0146] This path determination closed curve is delivered from the line thinning section 61 to the feature point deter- 
5 mining section 22. 

[01471 It is to be noted that the method of thinning the designation line is not limited to the above-described method 
but other methods may be used for this purpose. Namely, e.g., an approach may be employed to remove pixels con^ 
strtuting the designation line from pixels positioned outside to thereby genei^te a thin line having width consisting of 
one pixel to further determine a line which most satisfactorily approximate a line passing through the centers of pixels 
10 constituting that thin line to allow this line to be the path determination closed curve. 

[01 481 In the key signal generating apparatus constituted as described above, when designation line corresponding 
to the boundary between the foreground F and the background B (pixels constituting the boundary) as the result of the 
fact that the operation section 1 Is operated by user, thinning of that designation line is carried out at the line thinning 
section 61 of the feature point detecting section 60. whereby the path determination closed curve is determined. The 
path determination closed curve thus obtained is delivered to the feature point determining section 22. At times sub- 
sequent thereto, processing similar to the key signal generating apparatus shown in FIG. 10 Is earned out. Thus, a 
key signal a corresponding to the entirety of the pbture C is determined. 

[0149] Accordingly, since the path detemninatlon dosed curve Is determined on the basis of designation by user 
(designation line) in this case, change C* of the pixel value becomes unnecessary in order to determine the path 
detemiinatlon dosed curve. Namely, In this case, there is no necessity of delivering change C of the pixel value with 
respect to the feature point detecting section 60 from the change detecting section 1 0. 

[0150J While the key signal generating apparatus shown in FIG. 10 is adapted to detemiine difference F-B at the 
F-B detecting section 30 to normalize the change C of the pixel value thereafter to determine the key signal, the key 
signal generating apparatus may be of construction in which no F-B detecting section 30 is provided. 
[01 51 1 As a more practical example, the configuration of another embodiment of the key signal generating apparatus 
to which this invention is applied Is shown in FIG. 33. It is to be noted that, in the key signal generating apparatus 
shown in FIG. 33, the same reference numerals are respectively attached with respect to portions corresponding to 
the key signal generating apparatus shown in FIG. 10, and their explanation will be omitted as the occasion may 
demand. Namely, this key signal generating apparatus is adapted so that the F-B detecting section 30 is omitted and 
an integral value calculating section 70 is provided in place of the integral value cateulating section 40. In addition, 
other portions of the key signal generating apparatus are constituted In the same manner as in the key signal generating 
apparatus shown in FIG. 10. 

[0152) The integral value cabulating section 70 Is operative so that, within the range where quantity (magnitude) of 
change C* of the pixel value Is above a predetermined threshold value K (hereinafter refen-ed to as integral range as 
the occasion may demand), it integrates its change quantity, e.g.. along the direction in which change of the pixel value 
C has been detected, or the like to calculate a key signal (the skeleton portion of the key signal). 

[01 53) Namely, the integral value calculating section 70 first detemiines points on the boundary of the gray area., i. 
e.. points p,. pg where a is equal to 0, 1 of points on. e.g., the maximum inclination (gradient) direction path U (It Is to ' 
be noted that the path itself may be path except for the maximum inclination (gradient) direction path U as described 
above). In more practical sense, as shown in FIG. 34A. for example, the integral value calculating section 70 detemiines 
two points where magnitude of change C of the pixel value delivered from the change detecting section 1 0 becomes 
equal to the predetermined threshold value K to allow them to be respectively points p^ , pj (for example , when observing 
the change C of the pixel value from the background B side towanj the foreground F side, the point where the change 
C of the pixel value becomes equal to the threshold value K in the beginning is caused to be the point p^ , and the point 
where the change C of the pixel value becomes equal to the threshold value K thereafter is caused to be the point pj.). 

[0154) Further, the integral value cateulating section 70 catoulates, in accordance with the following equation (19), 
integral value (definite integral value) 0 (definite integral information) of change of the pixel value C* within the range 
(Integral range) from the point to the point pj. e.g.. along the maximum indination (gradient) direction path U. 
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[0155] Then, the Integral value cateulating section 70 normalizes the change C* of the pixel value by the definite 
integral value D to thereby determine normalized value C'/O to integrate this nonmalized value CVD, e.g., in accordance 
with the following equation (20) to thereby determine a key signal o with respect to the pixel at the point (position) p. 
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«(p) = [~f-dp ^ (20) 



[01 56] The definite integral value D corresponds to the absolute value IF-B1 of difference F-8 In the case of the key 
signal generating apparatus shown in FIG. 1 0. Accordingly, when the change C of the pixel value is normalized by the 
definrte integral value D to carry out the integration by the equatlon(20), ft is possible to easily obtain a key signal a 
which takes value within the range from 0 to 1 . It is to be noted that while, in this case, the key signal a is obtained 
within the range from the points (positions) pi to p2 (the portion labeled "effective (valid) width" in FIG. 34B), values 
of key signals within the range at the left side (positioned) relative to the point p^ or the range at the right side (positioned) 
relative to the point P2 in FIG. 348 are caused to be respectively 0 and 1 (interpolation of 0 and 1 is carried out at the 
key signal generating section 50 as described above). 

[01 57] Since it is unnecessary to provide the F-B detecting section 30 at this key signal generating apparatus from 
facts as described above, the apparatus can be of a smaller-size configuration as compared to the key signal generating 
apparatus shown in FIG. 10. 

[01 58] The operation of the key signal generating apparatus shown In FIG. 33 will now be described with reference 
to the flowchart shown in FIG. 35. 

[01 59] At step S21 , the change detecting section 1 0 calculates change C of the pixel value between pixels of picture 
(input picture) C similarty to the case of the step SI of FIG. 11 to deliver it to the feature point detecting section 20 and 
the integral value calculating section 70. 

[0160] At step S22, the feature point detecting section 20 is operative so that when the change C of the pixel value 
is delivered thereto from the change detecting section 1 0. it detects tiie path detenmination closed curve and the feature 
points similarty to the case at the step S2 of FIG. 11 to delwer the path determination closed curve and the feature 
points to the integral value calculating section 70. 

[0161] At step S23, the Integral value calculating section 70 is operative so that the change C* of the pixel value is 
delivered thereto from the change detecting section 10 and the path determination closed curve and the feature points 
are delivered thereto from ttie feature point detecting section 20, it detemiines the integral path, e.g., similarty to the 
integral value calculating section 40 of FIG. 1 0 to further detemnine points p^ , pg where magnrtude of change C of the 
pixel value becomes equal to a predetermined threshold value K. as has been explained with reference to FIG. 34A. 
on the integral path. Then, the integral value cateulating section 70 calculates integral value D of the quantity of the 
change C of the pixel values at the points on the integral path existing within the range from ttie point p. to the point 
P2 in accordance with the equation (19). 

(01 62] At step S24, the integral value cateulating section 70 noonalizes the change C of the pixel value by the integral 
value D to cateulate normalized value C7D. 

[0163] At step S25, the integral value calculating section 70 determines, in accordance with the equation (20), the 
key signal with respect to tiie points p constituting the integral path (path segment), i.e., the skeleton portion of the key 
signal to deliver it to the key signal generating section 50. Further, the Integral value calculating section 70 generates 
P1 closed curve and P2 closed curve similarty to the integral value calculating section 40 of FIG. 1 0 to deliver them to 
the key signal generating section 50. 

[0164] At step S26. tiie key signal generating section 50 canies out processing similar to that in the case of the step 
S6 of FIG. 11 to thereby generate a key signal a corresponding to the entirety of the picture C. Thus, the processing 
at ttie key signal generating apparatus is completed. 

[0165] An example of the configuration of the integral value calculating section 70 of FIG. 33 is shown in FIG. 36. 
[0166] Path determination closed curve and feature points from the feature point detecting section 20 are delivered 
to an integral path determining section 71, and change C of the pixel value from the change detecting section 10 is 
delivered to the integral path determining section 71 , a definite integral value cateulating section 72, and a normalization 
executing section 73. In this example, the integral path detemiining section 71 is also supplied with a predetermined 
threshold value K from circuit (not shown). 

[0167] The integral path detennining section 71 determines integral patiis passing through respective feature points, 
similarly to the integral path determining section 41 shown in FIG. 24, on the basis of the feature points and the path 
determination dosed ounces to deliver these integral paths to tiie definite integral value calculating section 72, the 
nomialization executing section 73 and ttie integration executing section 74. Further, the integral path determining 
section 71 determines points where \he magnitude of the change C of the pixel value becomes equal to the predeter- 
mined threshold value K, i.e., points p^, P2 of points on the integral paths passing through respective feature points to 
deliver these points p^. pg to the definite integral value cateulating section 72, an integration executing section 74 and 
a ctesed curve generating section 75. 
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[01 68] The definite integral value calculating section 72 performs an operation in accordance with the equation (19) 
Namely, the definite integral value calculating section 72 canies out definite integral of change C of the pixel value 
within the range from the point p^ up to the point pg similarly delivered from the integral path detemiining section 71 
along the Integral paths passing through respective feature points delivered from the integral path detemiining section 
5 71 to deliver, to the normalization executing section 73. integral values (definite integral values) 0 every respective 
integral paths obtained as the result thereof. 

[01 69] The normalization executing section 73 nomnalizes change C of the pixel value on each integral path delivered 
from the integral path determining section 71 by the definite integral value D with respect to each integral path delivered 
from the definite integral value calculating section 72 to deliver the nonDalized value C70 obtained as the result thereof 
10 to the integration executing section 74. 

[0170J The integration executing section 74 performs an operation in accordance with the equation (20). Namely 
the integration executing section 74 carries out integration of the normalized value C'/D delivered from the nomializatiori 
executing section 73 along each integral path delivered from the integral path determining section 72 to output the 
skeleton portion of the key signal as the integral value on each integral path (path segment) obtained as the result 
IS thereof. It is to be noted that while the integration along each integral path at the integration executing section 74 is 
carried out according to the equation (20) in such a manner that the point p^ on each integral path delivered from the 
integral path determining section 71 is caused to be the initial (start) point, such integration may be also carried out in 
such a manner that the point Pg is caused to be the initial point, 

[0171] The closed curve generating section 75 is operative so that when points p^. pz on all integral paths are de- 
livered thereto from the integral path determining section 71 . it connects the points Pi to each other, and connects the 
points P2 to each other to thereby respectively generate P1 closed curve and P2 closed curve. 
[01 72] The operation of the integral value cateulating section 70 will now be described with reference to the flowchart 
of FIG. 37. 

[0173] At step S21-1 , the integral path detemiining section 71 canies out processing similar to that in the case of 
the step S4-1 of FIG. 26 to determine integral paths passing through the remarked feature points to deliver these 
integral paths to the definite integral value calculating section 72, the nonmalization executing section 73 and the inte- 
gration executing section 74. 

[01 74] At step S21 -2. the integral path determining section 71 detects points p^ , pg where the magnitude of change 
C of pixel value delivered from the change detecting section 10 becomes equal to the predetermined threshold value 
K of points on the integral paths passing through the remarked feature points to deliver these points p, , pg to the deflriite 
integral value calculating section 72, the integration executing section 74 and the closed curve generating section 75. 
[0175] At step S21-3, the definite integral value calculating section 72 is operative so that when the integral path 
with respect to the remarked feature point and points p^ , pg on the integral path are delivered thereto from the integral 
path detemiining section 71 , it carries out integration of change C of the pixel value within the range from the point p^ 
to the point pj along the integral path in accordance with the equation (1 9) to deliver definite integral value D obtained 
as the result thereof to the nomnalization executing section 73. 
[01 76] At step S21 -4. the nomialization executing section 73 nonmalizes the change C* of the pixel value of pixel on 
the integral path with respect to the remarked feature point by the definite integral value D delivered from the definite 
integral value cateulating section 72 to delwer normalized value C/D obtained as the result thereof to the integration 
40 executing section 74. 

[0177] At step S21 -6, the integration executing section 74 integrates, in accordance with the equation (20), the nor- 
malized value CVD delivered from the nomnalization executing section 73, with, e.g.. the point p^ delivered frorp the 
integral path determining section 71 being as the initial (start) point, along the integral path similarly delivered from the 
integral path detemilnating section 71 to calculate the key signal on the integral path, i.e., the skeleton portion of the 
45 key signal. 

(0178] At step S21-6, the integral path detenmining section 71 judges whether or not all of feature points from the 
feature point detecting section 20 have been caused to be the remarked feature point. In the case where the integral 
path determining section 71 has judged at this step S21-8 that all of feature points from the feature point detecting 
section 20 have not yet been caused to be the remarked feature point, the processing retums to the step S21 -1 . Thus, 

so the integral path detennining section 71 allows the feature point whch is not yet caused to be the remarked feature 
point to be newly the remartced feature point. At times subsequent thereto, processing as described above is repeated. 
[0179] On the other hand, in the case where the integral path determining section 71 has judged at the step S21-6 
that all of feature points have been caused to be the remarked feature point, i.e., integration of the equation (20) along 
ail the integral paths passing through respective feature points delivered from the feature point detecting section 20, 

ss whereby key signals on those integral paths (the skeleton portions of the key signals) have been determined, the 
processing proceeds to step S21 -7. 

[0180] At the step S21-7. the closed curve generating section 75 generates PI closed curve and P2 closed curve. 
Thus, the processing at the integral value calculating section 70 is completed. 
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[01 81 1 Namely, the closed curve generating section 75 connects the points to each other and connects the points 
P2 to each other, those points existing on all integral paths delivered from the Integral path determining section 71 to 
thereby respectively generate PI closed curve and P2 closed curve. 

[01821 is to be noted that the integration at the integration executing section 74 is carried out only with respect to 
the path segment of the integral path similarly to the case at the integration executing section 43 shown in FIG. 24, 
[01 83] FIG. 38 shows the configuration of an embodiment of a picture synthesis apparatus to which this invention is 
applied. 

[0184] Picture providing units comprise, e.g.. a video tape recorder, hard disc device or disc player, etc., and serve 
to reproduce a predetemdined picture to output it. In this example, picture T1 (first picture) in which the foreground and 
the background that the picture providing unit 81 reproduces are caused to be respectively F and B is delivered to a 
key signal generating unit 84 and a mixer 85. Thus, this picture T1 is caused to be the object picture for generating 
the key signal. Moreover, picture T2 (second picture) that the picture providing unit 82 reproduces is delivered to the 
mixer 85, at whrch the foreground F of the prcture T1 is synthesized with respect to the picture T2- Further, picture T3 
obtained by photographing or picking up image of only the background B of the picture T1 that the picture providing 
unit 81 reproduces is outputted from the picture providing unit 83. Namely, when, e.g., the picture T1 is assumed to 
be the picture image where a person is walking within a certain scene, the pfeture T3 in which image of that scene 
(background B) is picked up in the state where that person (foreground F) is excluded is outputted from the picture 
providing unit 83. The picture T3 that the picture providing unit 83 reproduces is delivered to the mixer 85. 
[0185] The key signal generating unit 84 is constituted in a manner similar to the key signal generating unit of FIG. 
10, 30 or 33, and generates a key signal a as described above to deliver it to the mixer 85. The mixer 85 synthesizes 
the prcture T1 and the picture T2 inputted thereto by using the key signal a from the key signal generating unit 84. 
Namely, the mixer 85 extracts the foreground F from the picture T1 to synthesize (incorporate) that foreground F into 
(with respect to) the pteture T2. 

[01 86] The operation of the picture synthesis apparatus will now be described. 

[0187] At the picture providing units 81 to 83, ptetures T1 to T3 are respectively reproduced. These pictures T1 to 
T3 are delivered to the mixer 85. Further, the picture T1 that the picture providing unit 81 reproduces is delivered also 
to the key signal generating unit 84. 

[0188] The key signal generating unit 84 generates a key signal a in a manner as described above to deliver it to 
the mixer 85. The mixer 85 extracts the foreground F from the picture T1 inputted thereto by using the key signal a to 
further synthesize that foreground F Into (with respect to) the picture T2 to generate picture in whfch the foreground F 
is synthesized Into (with respect to) the ptoture T2 (synthetic picture) T4. 

[0189J Namely, assuming now that pixel values of pixels at respective positions p constituting respective pictures 
T1 , T2 are respectively designated at P(p), Q(p), pixel values of pixels at respective positions p constituting respective 
foreground and background of the picture T1 are respectively designated at F(p), B(p). and pixel value of pixel at each 
position p constituting the synthetic picture T4 is designated at R(p), the following equations (relational expressions) 
(21). (22) hold: 



P(P) = aF(p) + {1-a)B(p) 



(21) 



40 



R{p) = oF{p) + (1.a)Q(p) 



(22) 



[0190] When aF(p) of the right side of the equation (22) is eliminated from the equation (21), the equation (22) Is 
45 rewritten into the following equation (23) 

R(P) = P(P) - (1 -a)B(p) + (1 -a)Q(p) (23) 



[0191] At the mixer 85, synthetic picture T4 is generated in accordance with the equation (23). Namely, pixel values 
P(P). Q(P). B(p) are respectively delivered from the picture providing units 81 to 83 to the mixer 85, and the key signal 
a(p) is delivered from the key signal generating unit 84 also to the mixer 85. At the mixer 85, operation of the equation 
(23) is performed by using the respective pixel values and the key signal. Thus, picture T4 consisting of pixel value R 
(p). I.e., in which the foreground F is synthesized with respect to the picture T2 is generated. 
[0192] This picture T4 is delivered to. e.g., recording unit (e.g., video tape recorder, hard disc device, or writable disc 
drive, etc.). at whfch it is recorded. 

[0193] Since precise key signals, i.e., key signals in whfch the influence of motion blur is taken into consideration 
and alias is permitted to be reduced are outputted from the key signal generating unit 84 as described above, in the 
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case where such key signals are used to carry out synthesis of picture, precise synthetic picture (picture free from 
disagreennent of feeling when viewed) can be obtained. 

[0194] It is to be noted that the picture synthesis apparatus shown in FiG. 38 may be constituted as described below 
Namely, since background 8 is provided by the picture providing unit 83, in the case where the key signal generating 
unit 84 is constituted by such a key signal generating apparatus as shown in FIG. 10 or 30 including the F-8 detecting 
section 30, background B is delivered to the key signal generating unit 84 as indicated by dotted lines in FIG 38 thus 
making it possible to determine difference F-B from the background B in the key signal generating unit 84 
[01951 Meanwhile, since picture T2 consisting of pixel value Q(p) is obtained from the picture providing unit 82 and 
the key signal a is obtained from the key signal generating unit 84. if the pixel value F(p) of the foreground F can be 
recognized (the pixel value F(p) can be estimated), it is possible to generate, also by the equation (22). pteture T4 in 
which the foreground F is synthesized with respect to the picture T2. 

[0196] In view of the above, the configuration of an embodiment of the picture synthesis apparatus adapted for 
generating picture T4 in accordance with the equation (22) is shown in FIG. 39. It is to be noted that the same reference 
numerals are respectively attached with respect to portions corresponding to the picture synthesis apparatus shown 
in FIG. 38 m the picture synthesis apparatus shown in FIG. 39. and their explanation will be omitted as the occasion 
may demand. Namely, in this picture synthesis apparatus, an infomiation generating unit 90 and a mixer 96 are provided 
in place of the picture providing unit 83 and the mixer 85, In addition, other portions of the picture synthesis apparatus 
are constituted similarly to the pkJture synthesis apparatus of FIG. 38. 

[0197] The infomiation generating unit 90 is supplied with picture T1 that the pteture providing unit 81 outputs. The 
infomiation generating unit 90 estimates (generates), from the picture T1 . pixel value F(p) of the foreground F thereof 
to deliver it to the mixer 96. The mixer 96 perfonns an operation in accordance with the equation (22) by using pictures 
T1 , T2 (pixel values P(p). Q(p)) that the picture information providing units 81 , 82 respectively output, key signal a(p) 
that the key signal generating unit 84 outputs, and foreground F (pixel value F(p)) that the Informatiori generating unit 
90 outputs to thereby generate synthette ptoture T4 in whteh the foreground F Is synthesized with respect to the picture 
25 T2. 

[01 98] In the picture synthesis apparatus constituted as described above, pictures T1 . T2 are respectively reproduced 
at the pteture providing units 81 , 82. These pfctures T1 . T2 thus reproduced are delivered to the mixer 96. Further, the 
picture T1 that the picture providing unit 81 reproduces is delivered also to the information generating unit 90 and the 
key signal generating unit 84. The key signal generating unit 84 generates the key signal a in a manner as described 
30 above to deliver it to the mixer 96. 

[0199] On the other hand, the information generating unit 90 detemiines. from the picture T1 , pixel value F(p) of the 
foreground F thereof. 

[0200] A more practical example of the configuration of the infomiation generating unit 90 is shown in FIG. 40. This 
infomiation generating unit 90 is composed of a change detecting section 91 , a boundary detecting section 92, a very 
small area detecting section 93, a color distribution forming section 94, and a center of gravity detecting section 95. 
The change detecting section 91 is constituted similarty to the change detecting sections 10 respectively shown in 
FIGS. 1 0, 30 and 33, and the boundary detecting section 92 is constituted similarty to tiie boundary detecting section 
21 of FIG. 16. Moreover, ttie very small area detecting section 93, the color distribution forming section 94 and the 
center of gravity detecting section 95 are constituted respectively similarly to the very small area detecting section 31 . 
the color distribution forming section 32F and the center of gravity detecting section 33F. It is to be noted that the very 
small area detecting section 93 extracts only pixels constituting the foregrounds F with respect to respective very small 
areas to deliver pixel values of those pixels, i.e., only F boundary pixel values to the color distribution fonning section 94. 
[0201 ] Accordingly, at this infonnation generating unit 90. change P of the pixel value P of picture T1 is detemiined 
at the change detecting section 91, and path determination curve and feature points are determined on the basis of 
the change F at the boundary detecting section 92. Further, at the very small area detecting section 93. very small 
area including respective feature points is detected, and F boundary pixel values are extracted with respect to respective 
very small areas. Further, color distribution on the color space of F boundary pixel values in respective very small areas 
is determined at the color distribution forming section 94. In addition, center of gravrly of the color distribution is deter- 
mined at the center of gravity detecting section 95, and the center of gravity thus obtained is outputted as pixel value 
50 F(p) of the foreground F of pixels existing at the periphery of respective feature points in the gray area. 

[0202] In this example, at tiie information generating unit 90, with respect to the portion except for tiie gray area of 
the foreground F (completely the portion of the foreground F), pixel value F(p) of the foreground F thereof is outputted 
as it is. 

[0203] Tuming back to FIG. 39, at the mixer 96. pixel value R(p) of picture T4 In which the foreground F is synthesized 
with respect to the picture T2 Is determined in accordance with the equation (22) by using pixel values P(p). Q(p) that 
the picture infomiation providing units 81 , 82 respectively output, key signal a(p) that the key signal generating unit 84 
outputs, and pixel value F(p) that the information generating unit 90 outputs. 
[0204] Accordingly, also in this case, it is possible to obtain precise synthetic picture. 
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[0205] In this instance, at the infonnnation generating unit 90. background 8 may be generated in place of generating 
foreground F. In this case, at the mixer 96. pixel value R(p) of synthetic picture T4 may be determined in accordance 
with the equation (23) by using the background B generated by the infomnation generating unit 90. 
[0206] It is to be noted that the synthesis method for pteture at the mixers 85. 96 are not limited to the above-described 
method, but other synthesis methods may be used. 

[02071 Further, while integration is canried out at the integral value calculating sections 40. 70 In this embodiment, 
there may be employed, in place of the integration, an operation equivalent to the integration, i.e.. an operation to set 
sample points at the path for carrying out integration to add sample values at respective sample points. 
[0208] In addition, while key signal portion except for the skeleton portion of key signals in the gray area is determined 
by interpolation in this embodiment, key signal portion except for the skeleton portion of the key signal may be deter- 
mined, e.g.. by carrying out integration based on the equation (18) or the equation (20) in place of the interpolation. 
[02091 should be noted that various modifications and/or application examples are conceivable (applicable) within 
the range which does not depart from the gist of this invention. Accordingly, the gist of this invention is not limited to 
the embodiments. 

Industrial Applicability 

[0210] In accordance with the key signal generating apparatus and the key signal generating method according to 
this invention, change of pixel value between pixels constituting pk:ture Is detected to generate a key signal on the 
basts of change of that pixel value. Accordingly, precise key signal can be obtained. 

[0211J In accordance with the picture synthesis apparatus and the picture synthesis method according to this inven- 
tion, change of pixel value between pixels constituting a first picture is detected to generate a key signal on the basis 
of the change of that pixel value. Then, that key signal is used to synthesize the foreground picture of the firet picture 
and the background picture of a second picture. Accordingly, precise synthetic picture can be obtained. 



Claims 

1 . A key signal generating apparatus adapted for generating a key signal from a picture, 

the apparatus comprising: 

change detecting means for detecting a change of pixel value between pixels constituting said picture, and 
outputting change data expressing the distance on a color space between pixels dose to each other of said 
picture; and 

key signal generating means for generating said key signal on the basis of said change data integrated along 
a predetenmined integral path in a direction in which a change of the pixel value has been detected by the 
change detecting means. 

2. A key signal generating apparatus as set forth in claim 1 , wherein the key signal generating means Integrates said 
change data along said predetermined integral path within an integral range which is the range where said change 
data takes a predetemnlned threshold value or more to thereby generate the key signal. 

3. A key signal generating apparatus as set forth in claim 1 or 2, wherein 

the picture consists of foreground picture and background picture , and wherein 

the apparatus further comprises difference detecting means for detecting difference between the foreground 
picture and.the background picture to output difference data indicating that difference. 

4. A key signal generating apparatus as set forth in any one of clainns 1 to 3, wherein the key signal generating means 
comprises: 

normalizing means for nomializing the change data by the difference data to output nomnalized data obtained 
as the result of the nonnalization, and 

integration executing means for integrating the nonnalized data along said predetennined integral path. 

5. A key signal generating apparatus as set forth in any one of claims 2 to 4, wherein the key signal generating means 
comprises: 
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range detecting means for detecting the integral range; 

definite integral information generating means for integrating the change data along said predetermined inte- 
gral path within the integral range to generate definite integral information of the change data within the intearal 
range; ^ 

nonnalizing means for normaMzing the change data by the definite integral information to output normalized 
information obtained as the result of the normalization; and 

integration executing means for integrating the nonmalized information to thereby generate the key signal. 

A picture synthesis apparatus adapted for receiving a first picture consisUng of at least foreground picture and 
background picture . and a second picture consisting of at least background picture to synthesize the foreground 
picture of the first pk:ture and the background picture of the second picture, 
the apparatus comprising: 

change detecting means for detecting a change of pixel value between pixels constituting said first picture 
and outputting change data expressing the distance on a color space between pixels close to each other of 
said first picture; 

key signal generating means for generating said key signal on the basis of said change data integrated along 
a predetermined integral path in a direction in which a change of the pixel value has been detected by the 
change detecting means; and 

synthesis means for synthesizing the foreground picture of the first picture and the background picture of the 
second pfcture by using the key signal generated by said key signal generating means. 

7. A picture synthesis apparatus as set forth in claim 6. which further comprises difference detecting means for de- 
tecting difference between the foreground picture and the background picture of the first picture to output difference 
information indicating the difference. 

8. A picture syntiiesis apparatus as set forth in claim 6 or 7. wherein the key signal generating means integrates the 
change data within an Integral range which is the range where the change Information takes a predetemiined 
threshold value or more. 
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9. A key signal generating method for generating a key signal from a picture, 

the method comprising the steps of: 

detecting a change of pixel value between pixels constituting said picture, and outputting change data ex- 
pressing the distance on a color space between pixels close to each other of said picture is calculated; and 
generating said key signal on the basis of said change data integrated along a predetemiined integral path in 
a direction in which a change of ttie pixel value has been detected by said detecting step. 

10. A key signal generating method as set forth in claim 9. wherein said change data are integrated along said pre- 
determined integral path within an integral range which is the range where said change data takes a predetermined 
threshold value or more to thereby generate the key signal. 

11. A key signal generating method as set forth in claim 9 or 1 0. 

wherein the picture.conslst3 of foreground picture and background picture, 
the method comprising the step of 

detecting difference between the foreground picture and the background picture to output difference data 
indfcating that difference. 

12. A key signal generating method as set forth in any one of claims 9 to 11 , the method comprising the steps of: 

nonnalizing the change data by the difference data to output normalized data obtained as the result of the 
nomialization, and 

integrating the nomnalized data along said predetemiined Integral path. 

13. A key signal generating method as set forth in any one of claims 10 to 12. wherein the method comprising the 
steps of: 
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detecting the integral range; 

integrating the change data along said predetermined integral path within the Integral range to generate deHnite 
integral information of the change data within the integral range; 

nonnalizing the change data by the definite Integral information to output normalized information obtained as 
the result of the nomiallzation; and 

integrating the normalized infomnatlon to thereby generate the key signal. 

14. A picture synthesis method of receiving a first picture consisting of at least foreground picture and background 
picture, and a second picture consisting of at least background picture to synthesize the foreground pkiture of the 
first picture and the background picture of the second picture. 

the method comprising the steps of: 

detecting a change of pixel value between pixels constituting said first picture, and outputting change data 
expressing the distance on a color space between pixels close to each other of said first picture- 
generating said key signal on the basis of said change data integrated along a predetermined integral path in 
a direction in which a change of the pixel value has been detected by the detecting step; and 
synthesizing the foreground picture of the first picture and the background picture of the second ptoture by 
using the key signal generated by said generating step. 

15. A picture synthesis method as set forth In claim 14, 

the method comprising the step of 

detecting a difference between the foreground picture and the background pteture of the first pcture to output 
difference information indk^ating the difference. 

16. A pfcture synthesis method as set forth in claim 14 oris, wherein said change data are integrated along said 
predetenmined integral path within an integral range which is the range where said change data takes a predeter- 
mined threshold value or more to thereby generate the key signal. 



Patentansprtiche 

1 . Schlusselsignal-Erzeugungsvorrichtung zur Erzeugung eines SchlQsselsignals von einem Biid. 

wobei die Vorrichtung aufweist, 

eine Anderungs-Detektionseinrichtung zum Detektieren einer Pixelwertanderung zwischen das Bild darstel- 
lenden Pixein und Ausgeben von den Abstand auf einem Farbraum zwischen zueinander nahen Pixein des 
Bildes ausdruckenden Anderungsdaten, und 

eine Schlusselsignal-Erzeugungseinrichlung zur Erzeugung des Schlusselstgnats auf der Basts der entlang 
einem vorbestimmten Integralpfad in einer Richtung, in der von der Anderungs-Detektionseinrichtung eine 
Anderung des Pixelwertes detektlert worden ist, Integrierten Anderungsdaten. 

2. Schlusselsignal-Erzeugungselnrichtung nach Anspnjch 1 , wobei die Schfusselsignal-Erzeugungseinrichtung zur 
Erzeugung des SchlQsselsignals die Anderungsdaten entlang dem vorbestimmten Integralpfad In einem Integral- 
bereich integriert, welcher der Bereich ist, bei dem die Anderungsdaten einen vorbestimmten Schwellenwert Oder 
mehr annehmen. 

3. Schlusselsignal-Erzeugungsvonrichtung nach Anspruch 1 oder 2, wobei 

das Bild aus einem Vordergrundbild und einem Hintergrundbild besteht. und wobei 

die Vorrichtung au3erdem eine Differenz-Detektlonseinrichtung zum Detektieren einer Differenz zwischen 
dem Vordergrundbild und dem Hintergrundbild zum Ausgeben von diese Differenz anzeigenden Differenzda- 
ten aufweist. 

4. SchlOsselsignal-Erzeugungsvorrichtung nach einem der AnsprOche 1 bis 3, wobei die Schlusselsignal-Erzeu- 
gungsvomchtung aufweist: 

eine Normierungseinrichtung zur Nonmierung der Anderungsdaten durch die Differenzdaten zur Ausgabe von 
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als das Ergebnis der Normierung erhaltenen Normierungsdaten. und 

eine Integrations-Ausfuhrungseinrichtung zur Integration der normierten Dalen entlang dem vorbestimmten 
Integralpfad. 

5. Schlusse!signa|.Er7eugungsvorrichtung nach einem der Anspriiche 2 bis 4. wobei die Schlusselslanal-Erzeu- 
gungsvorrichtung aufweist: 

eine Bereich-Detektionseinrichtung zum Oetektleren des Integralbereichs, 

elne Bestimmtintegralinforniation.Erzeugungselnrichtung zur Integration der Anderungsdaten entlang dem 
vorbestimmten Integralpfad im Integralbereich zur Erzeugung einer Bestimmtintegralinfomiatlon der Ande- 
rungsdaten im Integralbereich. 

elne Nornilerungseinrichtung zur Nonnierung der Anderungsdaten durch die Bestimmtintegralinfomiation zur 
Ausgabe von als das Ergebnis der Normierung erhaltenen nonnierten Information, und 
eine Integrations-AusfQhrungselnrichtung zur Integration der normierten Information, urn dadurch das Schlus- 
selsignal zu erzeugen. 

6. Zum Empfang eines aus wenigstens einem Vordergrund- und einem Hintergrundbild bestehenden ersten Slides 
und eines aus wenigstens dem Hintergrundbild bestehenden zweiten Bildes ausgebildete Bildsynthesevorrichtung 
zum Synthetisieren des Vordergrundbildes des ersten Bildes und des Hintergnjndbildes des zweiten Bildes. 

wobei die Vorrichtung aufweist: 

eine Andenjngs-Detektlonseinrichtung zum Detektieren einer Anderung eines Pixeiwertes zwischen das erste 
Bild darstellenden PIxeIn und Ausgeben von den Abstand eines Farbraumes zwischen zueinander nahen 
Pixein des ersten Slides ausdrucKenden Anderungsdaten, 

eine Schlusselslgnal-Erzeugungseinrichtung zur Erzeugung des Schlusselsignals auf der Basis der entlang 
dem vorbestimmten Integralpfad in einer RIchtung, in der von der Andeajngs-Detektlonseinrichtung eine An- 
derung des Pixeiwertes detektiert worden 1st, Integrierten Anderungsdaten, und 

eine Syntheseeinrichtung.zum Synthetisieren des Vordergrundbildes des ersten Bildes und des Hintergnjnd- 
bildes des zweiten Bildes durch Verwendung des von der Schliisselsignal-Erzeugungseinrichtung erzeuaten 
Schlusselsignals. 

7. Bildsynthesevorrichtung nach Anspnjch 6, mit einer Dlfferenz-Oetektionseinrichtung zum Detektieren einer Diffe- 
renz zwischen dem Vordergrundbild und dem Hintergrundbild des ersten Bildes zur Ausgabe einer die DIfferenzen 
anzelgenden Differenzinfonrtation. 

8. Bildsynthesevorrichtung nach Anspruch 6 Oder 7. wobei die Schliisselsignal-Erzeugungseinrichtung die Ande- 
rungsdaten entlang dem vorbestimmten Integralpfad in einem Integralbereich integriert. welcher der Bereich ist. 
bei dem die Andenjngsdaten einen vorbestinrimten Schwellenwert Oder mehr annehmen. 

9. Schlusselsignal-Erzeugungsverfahren zum Erzeugen eines SchlQsselsignals von einem Bild, 

wobei das Verfahren die Schritte aufweist: 

Detektieren einer Anderung eines Pixeiwertes zwischen das Bild darstellenden Pixein und Ausgeben von 
Anderungsdaten, die den berechneten Abstand auf einem Farbraum zwischen zueinander nahen Pixein des 
Bildes ausdrucken, und 

Erzeugen des Schlusselsignals auf der Basis der entlang einem vorbestimmten Integralpfad in einer RIchtung, 
in der beim Oetektionsschritt eine Anderung des Pixeiwertes detektiert worden ist, integrierten Andenjngsda- 
ten. 

10. Schlusselsignal-Erzeugungsverfahren nach Anspnjch 9, wobei zur Erzeugung des Schlusselsignals die Ande- 
rungsdaten entlang dem vorbestimmten Integralpfad in einem Integralbereich integriert werden, welcher der Be- 
reich Ist, bei dem die Andenjngsdaten einen vorbestimmten Schwellenwert Oder mehr annehmen. 

11. Schlusselstgnal-Erzeugungsveriahren nach Anspruch 9 Oder 10, 

wobei das Bild aus einem Vordergrundbild und einem Hintergrundbild besteht. wobei 
das Verfahren den Schritt eines 

Detektierens einer Differenz zwischen dem Vordergmndbild und dem Hintergrundbild zum Ausgeben von die- 
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se Difterenz anzeigenden Differenzdaten aufweisi. 

12. Schlusselsignalerzeugungsverfahren nach einem der AnsprOche 9 bis 11, wobel das Verfahren die Schritte auf- 
weist: 

Nomnieren der Anderungsdaten durch die Differenzdaten zum Ausgeben von als das Ergebnis der Norierung 
erhaltenen nonnierten Daten. und 

Integriereri der normierten Daten entlang dem vorbestimmten Integralplad. 

13. Schlusselsignal-Erzeugungsverfahren nach einem der Anspriiche 10 bis 12. wobei das Verfahren die Schritte 
aufweist: 

Detektieren des tntegralbereichs, 

Integrieren der Anderungsdaten entlang dem vorbestimmten Integralpfad im Integralbereich. urn Bestimmtin- 
tegralinfonnation der Anderungsdaten im Integralbereich zu erzeugen. 

Normieren der Anderungsdaten durch die Bestimmtintegralinformation zunri Ausgeben der als das Ergebnis 
der NomDierung erhaltenen nomilerten Infonmation, und 

Integrieren der normierten Information, urn dadurch das Schlusselsignal zu erhalten. 

14. Bildsynthesevertahren zum Erhalten eines aus wenlgstens einem Vordergrundbild und Hintergrundbild bestehen- 
den ersten Bildes und eines aus wenigstens einem Hintergrundbild bestehenden zweiten Bildes zum Synthetisie- 
ren des Vordergrundbildes des ersten Bildes und des Hintergrundbildes des zweiten Bildes, 

wobei das Verfahren die Schritte aufweist: 

Detektieren einer Anderung eines Pixelwertes zwischen das erste Bild darstellenden Pixein und Ausgeben 
von den Abstand auf einem Farbraum zwischen zueinander nahen Pixein des ersten Bildes ausdruckenden 
Anderungsdaten, 

Erzeugen des SchlQsselsignals auf der Basis der entlang einem vorbestimmten Integralpfad in einer Richtung, 
in der beim Detektionsschritt eine Anderung des Pixelwertes detektlert worden ist. integrierten Anderungsda- 
ten, und 

Synthetisieren des Vordergrundbildes des ersten Bildes und des Hintergrundbildes des zweiten Bildes durch 
Venwendung des beim Efzeugungsschritt erzeugten SchlQsselsignals. 

15. Blldsyntheseverfahren nach Anspruch 14, 

wobei das Verfahren den Schritt eines 

Detektierens einer Differenz zwischen dem Vordergrundbild und dem Hintergrundbild des ersten Bildes zum 
Ausgeben einer die-Differenz anzeigenden Differenztnfomiatlon aufweist. 

1 6. Blldsyntheseverfahren nach Anspruch 1 4 Oder 1 5, wobei zur Erzeugung des SchlQsselsignals die Anderungsdaten 
entlang dem vorbestimmten Integralpfad in einem Integralbereich Integriert werden, welcher der Bereich ist, bei 
dem die Anderungsdaten einen vorbestimmten Schwellenwert oder mehr annehmen. 



Revendicatlons 

1. Un dispositif de g^ndration de signal d'incrustation adapte pour gen^rer un signal d'incrustation k partir d'une 
image, 

le dispositif comprenant : 

des moyens de detection de changement pour d6tecter un changement de valeur de pixel entre des pixels 
constituant I'image, et 6mettre des donn^es de changement exprimant la distance dans un espace de couleurs 
entre des pixels de I'tmage proches les uns des autres; et 

des moyens de generation de signal d'incrustation pour g^nerer te signal d'incrustation sur la base des don- 
nees de changement int§grdes le long d*un chemin d'intdgration pr6d6termin6, dans une direction dans la- 
quelle un changement de la valeur de pixel a 6t4 ddtect6 par les moyens de detection de changement. 

2. Un dispositif de gdn^ration de signal d'incrustation selon la revendHsation 1 , dans lequel les moyens de generation 
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de signal d'incrustation integrent les donnees de changement le long du chemin d'integration predetermine a 
I interieur d'une plage d'integration qui est la plage dans laqueile les donnees de changement prennent une valeur 
de seuil predeterminee. ou plus, pour generer ainsi le signal d'incrustation. 

Un dispositif de generation de signal d'incrustation selon la revendicatlon 1 ou 2, dans lequel 

rimage consiste en une image de premier plan et une image de fond, et dans lequel 

le dispositif comprend en outre des moyens de detection de difference pour d6tecter une difference entre 

hmage de premier plan et I'lmage de fond, pour foumir en sortie des donndes de difference indiquant cette 

difference. 



Un dispositif de generation de signal d'incrustation selon Tune quelconque des revendications 1 k 3. dans lequei 
les moyens de g6n6ration de signal tfincnjstatlon comprennent : 

des moyens de nomialisation pour nomnaliser les donnees de changement par les donnees de difference de 
fapon ^ fournir en sortie des donn6es nomialis^es obtenues comme le r6sultat de la normalisation, et 
des moyens d'ex6cution d'int6gratlon pour int6grer les donnees noimalisdes le long du chemin d'lnt6aratlon 
pr6d6temiin6. ^ 

Un dispositif de g6n6ratlon de signal d'Incnjstation selon Tune quelconque des revendications 2^4, dans lequel 
les moyens de generation de signal tfincmstation comprennent : 

des moyens de detection de plage pour d6tecter la plage d'integration; 

des moyens de generation d'Infomnatlon tfintegrale definie pour integrer les donnees de changement le long 
du chemin ^integration predetermine k I'interieur de la plage d'integration, pour generer une infomiation d'in- 
tegrale definie des donnees de changement k I'interieur de la plage d'integration; 

des moyens de nomialisation pour normaliser les donnees de changement par I'inf ormation d'Integrale definie, 
pour foumir en softie une information nomialisee obtenue comme le resultat de la nortnallsation, et 
des moyens d'execution ^integration pour integrer Tinfonnation normalis6e pour generer ainst le signal d'in- 
crustation. 

Un dispositif de synthase d'image adapte pour recevoir une premiere image consistant en au molns une image 
de premier plan et une image de fond, et une seconde image consistant en au moins une image de fond, pour 
synthetlser I'lmage de premier plan de la premiere image et I'image de fond de la seconde image, 
le dispositif comprenant : 

des moyens de detection de changement pour detecter un changement de valeur de pixel entre des pixels 
constituant la premiere image, et pour fournir en sortie des donnees de changement exprimant la distance 
dans un espace de couleurs entre des pixels de la premiere image proches les uns des autres; 
des moyens de generation de signal d'incrustation pour generer le signal d'incrustation sur la base des don- 
nees de changement integr6es le long d'un chemin d'integration predetermine dans une direction dans laqueile 
un changement de la valeur de pixel a ete detecte par les moyens de detection de changement: et 
des moyens de synthfese pour synthetlser I'lmage de premier plan de la premiere image et limage de fond de 
la seconde image en utiiisant le signal d'incnjstation genere par les moyens de generation de signal d'incrus- 
tation 



Un dispositif de synthese d'image selon la revendication 6, comprenant en outre des moyens de detection de 
difference pour detector une difference entre I'image de premier plan et I'image de fond de la premiere image, 
pour foumir en sortie une Infonnation de difference Indiquant la difference. 

Un dispositif de synthtee d'image selon la revendication 6 ou 7. dans lequel les moyens de generation de signal 
d'incrustation intfegrent les donnees de changement d i'interieur d'une plage d'integration qui est la plage dans 
laqueile t'information de changement prend une valeur de seuil predetenminee, ou plus. 

Un precede de generation de signal d'incrustation pour generer un signal d'incrustation k partir d'une Image, 
le procede comprenant les etapes suivantes : 

on detecte un changement de valeur de pixel entre des pixels constituant I'image, et on foumit en sortie des 
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donnees de changement exprimant la distance dans un espace de couleurs entre des pixels de I'image proches 
ies uns des autres; et 

on genere le signal d'incrustation sur la base des donnees de changement integrees le long d'un chemin 
d'integration pred§tefmin6 dans une direction dans laquelle un changement de la valeur de pixel a ete d6tecte 
^ par t'etape de detection. 

1 0. Un precede de generation de signal d'incrustation selon la revendication 9, dans lequel Ies donnees de changement 
sont integrees le long du chemin d'int^gration predetennin6 & i'int6rieur d'une plage d'int^gration qui est la plage 
dans laquelle Ies donnees de changement prennent une valeur de seuii pr6d6temiin6e, ou plus, pour g6n6rer 

10 ainsi le signal d'incrustation, 

11. Un precede de g6n6ration de signal d'incrustation selon la revendication 9 ou 10, 

dans iequei I'lmage conslste en une image de premier plan et une image de fond, 
le proced6 comprenant I'^tape consistant k 

15 

detecter une difference entre rimage de premier plan et I'image de fond pour fournir en sortie des donnees 
de difference indiquant cette difference. 

12. Un precede de generation de signal d'incrustation selon rune quelconque des revendlcatlons 9 & 11 , le proc6d6 
^ comprenant Ies etapes sulvantes : 

on nomialise Ies donnees de changement par Ies donnees de difference pour foumir en sortie des donnees 
nomnaIis6es obtenues comme le r6sultat de la nonnaiisation, et 
on integre Ies donn6es nomialis6es le long du chemin d'integration predetermine. 

1 3. Un precede de generation de signal d'incrustation selon Tune quelconque des revendlcatlons 10^12, dans lequel 
le procede comprend Ies etapes suivantes : 
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on detecte la plage d'integration; 

on integre Ies donnees de changement le long du chemin ^integration pr6detemiine k I'interieur de la plage 
d'integration, pour generer une infomiation d'integrale definie des donnees de changenrient k I'interieur de la 
plage d'integration; 

on nonmalise Ies donnees de changement par I'information d'Integrale definie pour foumir en sortie une infor- 
mation nomnalisee obtenue comme le resultat de la normalisation; et 
^ on integre i'infomiation nomnalisee pour generer ainsi le signal d'incrustation. 

14. Un precede de synthese d'image dans lequel on report une premiere image consistant en au moins une image de 
premier plan et une Image de fond, et une seconde image consistant en au moins une Image de fond, pour syn- 
thetiser rimage de premier plan de la premiere image et I'image de fond de la seconde image. 

^ le precede comprenant Ies etapes suivantes : 

on detecte un changement de valeur de pixel entre des pixels constituant la premiere image, et on fournit en 
sortie des donnees de changement exprimant la distance dans un espace de couleurs entre des pixels de la 
premiere image proches Ies uns des autres; 
^ on genere le signal d'incrustation sur la base des donnees de changement integrees ie long d'un chemin 

d'integration predetemnine dans une direction dans laquelle un changement de la valeur de pixel a ete detecte 
par retape de detection; et 

on synthetise I'image de premier plan de la prem lere image et rimage de fond de la seconde image en utilisant 
le signal d'incrustation genere par i'etape de generation. 

so 

15. Un procede de synthese d'image selon la revendication 14. 

le precede comprenant retape suivante : 

on detecte une difference entre i'image de premier plan et I'image de fond de la premiere image pour fournir 
en sortie une infomiation de difference indiquant la difference. 

16. Un procede de synthese d'image selon la revendication 14 ou 15. dans lequel ies donnees de changement sont 
integrees le long du chemin d'integration predetennine k i'interieur de la plage d'integration qui est la plage dans 
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laquelle les donn6es de changement prennent une valeur de seuil predeterminee. ou plus, pour gene 
signal d'incrustation. 
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